Research Productivity of Australian Planning Academics: A Bibliometric Analysis

Abstract

Spanish

Chinese

The development of global metrics for evaluating university research performance has been accompanied by increasing attention to key performance metrics for individual disciplines. This paper examines research performance metrics for Australian planning academics. It addresses questions related to programs, staff, publications, and citations. The main findings are the following: Wide gender gaps exist in Australian planning academia; the mean number of publications is 36 per person, or 3 per person per year; the mean number of citations is 527 per person, or 48 per person per year; and planning journal impact factors are low (less than 3).

Keywords

citation analysis bibliometrics research performance planning academia Australia

Introduction

Global universities now operate within a framework that some scholars have described as “neoliberal”—the competitive pursuit of efficiency and excellence in the drive for prestige and increased income (Batterbury and Byrne 2017). Accompanying this shift, the evaluation of university research performance based on key metrics, including publications and citations, has become a global practice. Four rankings—the Shanghai Jiao Tong University Academic Ranking of World Universities, the Times Higher Education and QS World University Rankings, and CWTS Leiden Ranking—now exert considerable influence on academic research investment and funding decisions (Hicks 2012). Individual universities, departments, and scholars are under increasing pressure to improve their standing on the world stage (Jöns and Hoyler 2013; Taylor and Braddock 2007).

In this article, we benchmark research performance metrics for Australian planning academics, investigating the impact of the global shift to performance-based assessment. Australian universities, like many international counterparts, are now funded under competitive, performance-based funding arrangements (see Batterbury and Byrne 2017). While comprehensive assessments of publication and citation metrics exist for the United States (see Sanchez 2016, 2017; Stiftel, Rukmana, and Alam 2004), less is known about other centers of planning scholarship, presenting an important knowledge gap. Our research has been guided by the primary question, “In a performance-based environment of benchmarking research excellence, what factors account for differences in research productivity (publications, citations, and grants) among Australian planning scholars?” A key finding is how the combination of historical patterns of gendered labor in the Australian academy (mirroring those internationally) may be combining with contemporary performance-driven indicators of publication, citation, and grant winning to disadvantage less-senior female academics.

As a discipline, planning research productivity is oftentimes unflatteringly compared with the sciences. Sanchez (2014, 2016, 2017) for example, has shown that science-based aggregators of publications and citations (e.g., Web of Science, Scopus) report fewer publications and citations than Google Scholar. Planners may be disadvantaged if hiring and tenure committees rely on the former rather than the latter. This is because Google Scholar also includes “nontraditional” publications (e.g., reports) outside scholarly sources. Although such “gray” literature is seldom peer reviewed, its inclusion in citation metrics reflects its greater reach and impact compared to pay wall–protected publications. Our research therefore compares Google Scholar and Scopus metrics. Google Scholar is most commonly used by Australian planning academics when applying for tenure, promotions, or grants.

The paper reports findings against four core areas:

Programs: How many planning programs are there in Australia?

Staff: What is the distribution of planning academics among ranks, and are there gender differences in this distribution?

Publications: How much “scholarship” do planning academics produce? What variances are there by institutional standing (e.g., membership of Group of Eight [Go8] universities¹), and why? How does productivity change as academics progress through their careers? Are there differences by gender? What impact factors (IFs) do the most popular peer-reviewed planning journals have?

Citations: How often is the work of planning academics cited? What variances are there by institutional standing (e.g., membership of Go8 universities) and why? How does impact (citation counts) change as academics progress through their careers? Are there differences by gender? How does Australia compare to major centers of planning scholarship, such as the United States?

Benchmarking research performance is important not only because it can affect an individual’s opportunities for tenure and promotion, and mental well-being, but because judgments made on research performance can impact departmental income, morale, and reputation. This matter is especially salient for Australian planning programs, because they tend to be small and are often part of large interdisciplinary schools, which can include architecture, geography, political science, information technology, engineering, and/or environmental science. Planners can find themselves disadvantaged if faculty administrators do not recognize that these disciplines apply different metrics to the “measurement” of academic performance.

Background

In a hypercompetitive global education market, research productivity and performance are seen as important ways to distinguish between the relative standing of different institutions; they have become proxy measures for the quality of teaching and course offerings (Byrne 2017). The use of “performance metrics” has emerged as the principal way to assess “quality.” For planning academics, the main forms of research output include books, book chapters, and journal articles but also professional reports, conference papers, studio/workshop projects, blog entries, and other electronic media (Sanchez 2017). Nearly all these products involve the retrieval and review of previously produced works as part of the process of creating new knowledge. Relevant items are cited for their authority and/or context (including examples of alternative ideas or conclusions). This process generates a network of citations that can be both quantitatively and qualitatively evaluated. With the advent of the Internet, citation data can easily be retrieved online from indexes, such as Scopus, Web of Science, and Google Scholar.

It is important however, to recognize that the practice of ranking academics based on research performance metrics (e.g., number of publications, journal IFs, citations, research income), is peculiar to the present time and is not ideologically “neutral.” Metrics are seldom undergirded by intellectual, political, or moral theory; they carry a strong normative bias. As Neil Postman (1993: p. 13) noted in Technopoly, embedded in such numbers is “a predisposition to construct the world as one thing rather than another, to value one thing over another, to amplify one sense or skill or attitude more loudly than another.” The use of metrics to rank people—first university students and now academics—is reshaping what is meant by quality.² Ranking practices can affect deeply embedded habits of thought and can privilege particular ontologies and epistemologies of knowledge production to the exception of others. Recent performance-based ranking practices, for example, represent a paradigm shift, ousting subjective forms of evaluation that have traditionally guided academic practice. If we simply accept the hegemony of numerical rankings in academia, scholars risk becoming, in Henry David Thoreau’s (1854) words, “tools of our tools.” Standardized metrics are reductionist and they obscure nuance, complexity, detail, and ambiguity in scholarly outputs. They provide only a limited range of formal, objective, and impersonal information. Yet many contemporary performance evaluation and promotion committees believe that without numbers, they cannot accurately assess an academic’s performance.

In Australia, as elsewhere, such metrics have rapidly become the normal way of assessing academic performance (Byrne 2017). They play a substantial role in both confirmation of tenure and application for promotion. Given the consequences for Australian universities of reduced federal funding from suboptimal performance (e.g., not being able to maintain research facilities, failing to attract international students), Australian scholars are generally under considerable pressure to “publish or perish” (Sanchez 2017). Australian academics are regularly counseled about how to cultivate increased citation of their publications (Innovative Research Universities 2018). This includes publishing in open-access journals, publishing alongside international scholars from prestigious institutions, maintaining an online Google Scholar profile, and participating in public engagement via social and mainstream media. Although in 2011 the Australian Research Council (ARC)—the national research funding agency—ceased rating peer-reviewed journals for the Excellence in Research for Australia assessment, such metrics have become institutionalized within managerialist business practices of Australian universities as they compete in a global education market (Byrne 2017; Connell 2015). Education is now one of Australia’s top exports, earning the country nearly AUD$30 billion annually (Dodd 2017), so demonstrating research excellence is “serious business.”

Practical Shortcomings of Academic Ranking and Metrics

While publication and citation counts, IFs, and h-indexes are said to represent the impact and influence of academic output over time, in additional to the conceptual inadequacies noted earlier, these metrics have recognized practical shortcomings. We briefly review these, before turning to the assessment of Australian planning research output. We do so because it is important to acknowledge limitations when undertaking this type of analysis.

Citation counts are either reported on their own or used to compute other metrics (e.g., IFs and h-indexes). Context is important. In planning and other branches of academia, nonscientific factors influence the decision to cite. Many citations are used simply to validate an article’s introduction, having no real significance to the rest of the work. Articles that focus on fashionable topics can improve citation chances. Considerable citation bias exists in favor of review articles. Well-established scholars are cited disproportionately more often than lesser-known scholars (so-called gratuitous authorship). Many authors tend to cite their own close colleagues more than others (i.e., “publication cartels”). Citations may oftentimes stem from negative or critical assessments (e.g., identifying flaws in prior research), yet this is not factored into metrics (Bornmann and Hans-Dieter 2008; Hyland 2016).

First devised in 2005, the h-index attempts to measure both the productivity and citation impact of the publications of a scholar. An h-index of 10, for example, indicates that an author has published 10 papers, each of which has been cited at least 10 times (Hirsch 2005). As citation counts, the h-index is also misleading. It is generally effective only for comparing researchers working in the same field—citation conventions differ widely among different fields. This is problematic for planning programs, which are embedded in multidisciplinary schools. Planning itself is so diverse that h-indexes can vary substantially (e.g., between urban transportation, health issues, or planning history). Moreover, the h-index fails to distinguish the relative contributions to the work in multiauthor articles and does not reflect the length of time an academic has been publishing. Junior or early-career researchers typically have lower h-indexes. Moreover, the h-index captures only part of a scholar’s publication and citation data; it fails to represent highly and rarely cited or noncited papers. Academics with very different citation frequencies can have the same h-index. For example, two academics with an h-index of 10 may each have 10 papers with 10 citations, but one may have an additional 90 papers with 9 citations each, which are unaccounted by the index; or one may have exactly 10 papers with 10 citations each and the other exactly 10 papers with 100 citations each (Bornmann and Hans-Dieter 2009). To correct for this, Google Scholar has recently introduced a complementary i10-index (the number of publications with at least 10 citations).

The Clarivate Analytics impact factor of an academic journal is a measure reflecting the annual average number of citations to recent articles published in that journal (e.g., in the last two years).³ Typically, it is used as a proxy for the relative importance of a journal within its field. Owing to its availability and utility, promotion committees, funding agencies, and scholars often also use it as a shorthand assessment of the quality of individuals or institutions. IFs were instituted as early as 1975 (initially as a way for librarians to make more informed decisions about journal subscriptions). However, IFs are misleading and can distort the communication of scientific progress due to selective attention to publications in high-ranking journals (Brembs, Button, and Munafò 2013).

When IFs are used to evaluate individual papers rather than journals, this devalues papers in subjects such as planning—as IFs of planning journals are comparatively low (e.g., 2–3) compared to science, technology, engineering, and mathematics disciplines (e.g., 50). As with the h-index, the IF is affected by field-dependent factors and cannot be used to compare journals across disciplines or even between planning subdisciplines (e.g., environmental planning and urban design). A high IF can be the skewed result of self-citation, within-journal citations, or many citations of just a few articles (e.g., review articles or articles led by well-known senior researchers) rather than the average level of the majority, reducing its value as an objective measure. Some commentators contend that self-citation is less of a concern for individual authors than for journal IFs. The latter are more susceptible to self-citation because relatively small numbers of citations can produce a significant change (Harzing 2010; Stevens 1990).

Commentators have also noted that many academics are now more concerned about publishing in high-IF journals than they are about the social impact of their research (Batterbury and Byrne 2017). Moreover, the practice of submitting articles to journals at the top of the IF ladder, circulating progressively through journals further down the rungs when they are rejected, can be a waste of time for both editors and reviewers (Campbell 2008; Simons 2008). Because not all journals are assigned an IF by Clarivate Analytics, publishers have begun to devise other indexes, including CiteScore, RIP (raw impact per paper), SNIP (source-normalized impact per paper), and FWCI (field-weighted citation impacts), which are more transparent and/or attempt to account for the differences in research behavior across disciplines. However, these esoteric metrics are also vulnerable to manipulation.

Alternative metrics

Planning is a professional discipline. Not only do academics generate scholarly research outputs, but they are also expected to contribute to professional practice, guide sustainable development, and connect directly with local communities through service and outreach (Krumholz 1986; Spain 1992; Wiewel, Carlson, and Friedman 1996). As such, planning research performance cannot be solely measured through standardized academic metrics (Sanchez 2014; Wachs 1994). Underscoring this limitation, alternative new metrics have emerged, such as “webometrics,” “altmetrics,” and measures used by Academia.edu and ResearchGate. These have the potential to capture and assess a broader range of scholarly impact than traditional citation analysis and bibliometrics (Aguillo 2012; Kousha and Thelwall 2008; Priem et al. 2010). Webometrics assess “informal” impact, which is primarily associated with educational impact (e.g., citations in course syllabi posted online), scholarly presentations (at conferences or seminars), and blog impact (Kousha, Thelwall, and Rezaie 2010). Altmetrics include measures of usage in addition to “formal” citations: downloads, views, shares on social media, bookmarks, expert reviews, user comments, and forum discussions (Kousha et al. 2010; Priem et al. 2010). Sites such as Academia.edu and ResearchGate serve as alternative repositories of academic products and employ their own metrics and analytics (e.g., online traffic overview and RG Score).

Alternative metrics are particularly important for planning academics given a range of gray literature (e.g., reports) they produce (Hurt and Yin 2006; Sanchez 2014). However, in practice, achieving high alternative scores requires academics to spend considerable time curating and updating social media profiles, self-promoting, and cultivating online personas. Assessing the impact of public service may be even more difficult than measuring traditional scholarship. Planning processes tend to be slow, and the impact of academic work might take years or decades to materialize. Moreover, positive urban change is rarely the outcome of solitary visionaries; it typically owes to the joint efforts of planning academics and practitioners (Checkoway 1997; Frank 2008).

Methods

We have employed similar methods to those used by Sanchez (2017) in his assessment of the U.S. planning academy. To answer the research questions we outlined in the introduction, we have used four sets of data about planning programs/academics in Australian universities: accredited programs, staff, publications, and citations. The study timeline was 2006 through 2016, a time period reflecting the implementation of performance assessment metrics in Australia (Innovative Research Universities, 2018). Since our data were collected in 2016, some planning academics have changed positions (parallel moves to a different university, promotions within the same university, or both a move and a promotion) or left Australia, and new academics have been recruited from abroad in some planning programs.

Programs

Data on the number of accredited planning programs offered by Australian universities were obtained from the Planning Institute of Australia (PIA). PIA is Australia’s professional association of planning practitioners and academics, and accredits planning programs. While undergraduate and graduate programs are noted separately, they are not different faculties, but they are different degrees. Unlike most universities in the United States, where planning qualifications are at the master’s (postgraduate) level, some Australian universities offer both undergraduate planning degrees and master’s (requalification) degrees.

Staff

Because most planning programs are embedded in interdisciplinary schools and their courses are taught by academics with a variety of backgrounds (many of whom do not self-identify as planners), ascertaining reliable numbers was difficult. A list of all academics affiliated with planning programs was created based on the information from university websites. That list included individuals that did not appear to have a strict planning background but taught into planning programs. Program coordinators and/or directors of all accredited planning programs were then contacted (via e-mail or telephone) to confirm the number of academics in their program, and the list was modified accordingly.⁴

In Australia there are five academic ranks: (1) Associate lecturer, (2) lecturer, (3) senior lecturer, (4) associate professor, and (5) professor.⁵ The metrics for staff members not on a traditional academic appointment (i.e., research fellows, senior research fellows) were also calculated.⁶ Teaching Fellows were excluded because they typically do not produce research.

Publications

For those academics appearing on the list, data were collected for the past decade, 2006 through 2016. A list of publications was generated from the staff profiles on university webpages, Google Scholar, and Publish or Perish and was then cross-referenced to ensure accuracy. Publication types included peer-reviewed journal articles, book chapters, edited books, authored books, and conference papers (both peer reviewed and non–peer reviewed as it was difficult to distinguish between the two). Publications and citations were then aggregated by (1) program, (2) publication type, (3) academic rank, and (4) gender (male/female only). Both totals and averages were computed; publication metrics were correlated with citation metrics. As part of this process, the top 20 journals in which planning academics publish along with their IFs were also identified.

Citations

Citation data were obtained via Google Scholar and Publish or Perish by visiting the profiles of each individual on the list of planning academics. At the time of writing, a surprising number of planning academics—71 out of 196 (36 percent)—did not have a Google Scholar profile. The distribution of those without a Google Scholar profile was about equal across ranks. It is possible that academics who are more productive, and whose work is more highly cited, are more likely to have public Google Scholar profiles. Also, younger academics may be more likely to curate their web presence. And several academics have transitioned to retirement since our data were collected and analyzed, or moved universities. For academics without a Google Scholar profile, publication/citation data were obtained through university website profiles and Publish or Perish.

Bibliometrics: Google Scholar versus Scopus

The same process, and for the same list of planning academics and the same time frame (2006–2016), was repeated to extract publication and citation from the Scopus database. About 6% of planning academics (12 in total) did not have as Scopus profile. In that case, publication/citation data were obtained through Publish or Perish. These data are compared, in aggregate form, in Table A-6 in the online supplemental material.

Results

We report our results and answer the research questions set forth at the outset using the same parameters identified in the methodology above (programs, staff, publications, and citations). In a final section on bibliometric analysis, we provide empirical evidence for our argument that in the case of planning, Google Scholar is preferable to other aggregators, such as Scopus.

Programs

How many planning programs are there in Australia?

A total of 48 accredited planning degrees at 24 universities were identified, staffed by 196 planning academics (Table A-1 in the online supplement).⁷ Planning programs are clustered along the southeast coast of Australia, where most large cities are located (Figure A-1 in the online supplement). Among the Go8 universities (the most prestigious universities in Australia, comparable to elite institutions in other countries), only six offer planning programs: University of Western Australia, University of Adelaide, University of Queensland, University of Melbourne, University of New South Wales, and University of Sydney. Monash University and Australian National University do not currently offer planning courses (although Monash is in the process of opening a postgraduate planning program).

Staff

What is the distribution of planning academics among ranks?

There are few associate lecturers in planning, as these appointments are typically made at pre-PhD level (Figure 1). Also, fewer than 10 percent of planning academics have research-only positions. Research-focused appointments are clustered in a few universities that have established planning research centers as part of an institutional strategy for improving research performance.⁸ No strong pattern can be discerned in the distribution of the remainder academic ranks (lecturer through professor). While there are more junior academics (lecturers and senior lecturers) than senior academics (associate professors and professors), the differences are not striking. This is partly explained by the fact that promotions in Australia are based on individual merit. The situation differs considerably in some European countries where rigid pyramidal hierarchies are maintained, with a single professor leading a research group. That system constitutes a significant barrier to the career progress of junior planning academics.

Figure 1.

Percentage of planning staff by rank.

What are the gender differences in the distribution of planning academics among ranks?

In regard to gender equity, the data reveal significant disparities (Figure 2). Overall, males outnumber females: 109 (56 percent) versus 87 (44 percent). Females are overrepresented in junior positions (lecturer ranks) but underrepresented in senior positions (professorial ranks). While women represent the majority in research-only positions, these are few, as noted, and not always tenured.

Figure 2.

Gender division by academic rank.

Publications

How much do planning academics produce?

Between 2006 and 2016, Australian planning academics produced a total of 7,038 publications, comprising journal articles, conferences papers, books, and book chapters. Overall, the median number of publications was 26.2 per person and 3 per person per year (Table A-2 in the online supplement).

What variances are there by institutional standing and why?

While having 31 percent of planning staff, Go8 universities contributed 39 percent of the total publications produced during the study time frame. The variances are noted in Table A-2 in the online supplement. Obviously, universities employing more planning academics have an advantage in total output even where individuals are not as productive. For example, the University of New South Wales, which has the most planning academics (21), also has the most total publications (1,065). However, the mean per academic is much smaller there than at the University of Melbourne (50.7 vs. 76.3), which has produced fewer publications in total (687) due to its smaller faculty size (only nine members). The University of Sunshine Coast has a high mean (49.3), despite having just three planning faculty members.

Because not all planning academics produce at the same rate, means can be misleading. In some cases, a few individuals produce a disproportionally large number of publications. This is especially true among the top 10 most productive schools. The most outstanding example is at Queensland University of Technology, where one academic has contributed more than 200 publications during the study period. Other universities where an individual academic has contributed 80 or more publications during the same period include University of New South Wales, Griffith University, University of Melbourne, and Western Sydney University. If these productive academics were omitted from the calculations, the annual and total means would be significantly lower. Only one university, University of Melbourne, ranks in the top five on all publication metrics: totals, means, and medians.

Among the top 10 universities in Table A-2 in the online supplement, 4 belong to the Go8: University of New South Wales, University of Melbourne, University of Queensland, and University of Western Australia. This finding suggests that, in the case of planning, it is important to look at the performance of individual programs rather than assume that a “brand name” university will automatically include a top planning program (as determined by productivity). From a student’s perspective, the scholarly reputation of academics is not the only criterion to consider when deciding where to study planning. In fact, teaching quality, curriculum offerings, and average class size might be more important to prospective students than the academic output of their instructors. This study did not examine whether these factors correlate with the quantity and quality of publications produced by planning academics, but Griffith University has recently been recognized as the top in the nation for student satisfaction.⁹

How does productivity change as academics progress through their careers?

On average, the number of publications increases as academics progress through the ranks (Figure 3). However, the progression is not linear; output tends to plateau once individuals reach the top rank of professor (see also Riordan 2011). If one looks at the mean number of publications, it is actually higher for associate professors than for professors, while the median is slightly lower. This discrepancy is most likely explained by the presence of a few highly productive associate professors who have not yet been promoted to professor or perhaps by increasing leadership service obligations at the highest rank.

Figure 3.

Publications by rank.

The difference in output between research fellows and senior research fellows is striking. While research fellows publish as much as lecturers, senior research fellows publish about the same as professors. This is partly explained by the nomenclature applied to research-only positions. Research fellows are often postdocs who are yet to start on the academic ladder, whereas senior research fellows are often longer-term, experienced staff members.

Are there differences by gender?

The differences in output between males and females are significant, as shown in Table 1. Overall, males produce about one third more than females. Part of this gender difference is due to the fact that women predominantly occupy lower academic positions. When men and women within the same rank are compared, a more complex picture emerges (see Figure A-2 in the online supplement). Looking at both means and medians, women publish slightly more than men at the lecturer level—at which they also outnumber men. If only medians are taken into account, women publish more than men at the associate professor level, too, which may mean that some have become “stuck” at that level and have not been promoted to professor. But men publish more than women at every other rank and based on all other metrics. The productivity of women falls slightly at professor level compared to associate professor level. Publication means and medians within the same gender also vary, especially among associate professors and professors. This suggests the presence of a few highly productive outliers in the professorial ranks, which are male dominated.

Table 1.

Publications and Citations by Gender.

Variable	Mean No. of Publications per Person	Median No. of Publications per Person	Mean No. of Citations per Person	Median No. of Citations per Person
Male	42	32	703	360
Female	28	22	308	148
Gender gap	33%	31%	56%	59%

Note: The gender gap was calculated based on the formula: (males – females)/males.

What IFs do the most popular peer-reviewed planning journals have?

The most popular journals where Australian planning academics have published are shown in Table A-3 in the online supplement, along with their IFs. Even though the list contains some of the top journals in the field, IFs are relatively low when compared with other disciplines. They range from 0.69 to 2.56, and many journals on the list, although highly regarded, do not have an IF. The full list of journals indicates substantial diversity in the field. In addition to traditional outlets, such as Cities and Environment and Planning, planning-related research has been published in journals as far apart as Journal of Epidemiology and Community Health (health focused), Ocean and Coastal Management (environment focused), Applied Mechanics and Materials (engineering focused), Tourism Review International (tourism focused), Progress in Human Geography (geography focused), and Gender, Place and Culture (gender focused). A recent study over approximately the same time period (using Scopus data) found that most academics have tended to publish in Australian Planner (official journal of PIA), reflecting a culture of trying to communicate research findings to practitioners (Byrne 2017).

Citations

How often is the work of planning academics cited?

The outputs of Australian planning academics have been cited 103,231 times between 2006 and 2016, as shown in Table A-4 in the online supplement. However, without comparable international benchmarks, we do not know whether this statistic is significant.¹⁰ Some key findings are provided below:

The median number of citations per person is 188.

The mean number of citations per person is 527.

The mean number of citations per person per year is 48.

The large difference between the mean and the median suggests the presence of a few academics whose work has attracted a significantly larger number of citations than the rest. As with the number of publications discussed above, in some cases a single faculty member has contributed a disproportionate number of citations.

What variances are there by institutional standing and why?

While constituting 31 percent of the total staff, planning academics in Go8 universities contributed 44 percent of the total citations during the study time frame. The median and mean citations in Go8 universities are 400 and 698, respectively. In other words, the Go8 median is more than twice as high as the overall median. The mean number of citations per person per year is 64.

The University of Melbourne and the University of Western Australia consistently ranked in the top five based on all citation metrics—although the University of Western Australia did not consistently rank in the top five based on publication metrics. As with publication counts, only 4 Go8 universities (out of 6 with planning programs) are among the top 10 in terms of total citations: University of New South Wales, University of Melbourne, University of Western Australia, and University of Queensland. Planning departments producing the most highly cited publications are in the top universities internationally, based on the Times of Higher Education index (see Table A-5 in the online supplement).

How does impact (citation counts) change as academics progress through their careers?

Citation counts increase as academics progress along their career, almost doubling from one academic level to the next (see Figure 4), unlike publications, which tend to plateau (Figure 5). As planning academics become more established and form a broader network, their papers tend to be cited more frequently. Another explanation might be that as academics become more experienced, they learn to subordinate their curiosity about particular research issues to “fashionable” topics, which are more likely to attract citations, or specialize in specific fields. But overall, at the university level, the relationship between publications and citations is relatively strong, as shown by a simple linear regression (Figure A-3 in the online supplement). Planning schools with more productive staff will attract more citations over time.

Figure 4.

Citations by rank.

Figure 5.

Metric trends by rank.

Are there differences by gender?

The same gender gaps seen in publications persist in citation metrics; the gaps are as pronounced (see Table 2 and Figure A-4 in the online supplement). The work of female planning academics tends to attract fewer citations than male academics. Patriarchal structures that undervalue women are a potential factor, and/or males may cite themselves (and each other) more often. Only at associate professor level (median only) do women appear to outperform men. This may mean that some highly performing women are “stuck” at this level and have not been promoted to professor. This is likely due to a failure to attract research grants—another important criterion for promotion in Australian universities. Grant winning may be a function of both male dominance in awarding agencies and of senior male academics being less encumbered by gendered domestic disparities (e.g., childcare).

Table 2.

Comparison with the United States.

U.S. University^a	Australian University	Median No. of Citations per Planning Academic
U.S. University^a	Australian University	United States	Australia
Columbia University^b	James Cook University	2,390	800
University of Minnesota^b	University of Melbourne^b	2,252	680
University of California, Los^b Angeles	University of Western Australia^b	1,994	471
Harvard University^b	University of Adelaide^b	1,577	382
New York University^b	University of New South Wales^b	1,551	363
Wayne State University	University of Queensland^b	1,340	361
University of Southern California^b	Queensland University of Technology	1,244	353
University of Toronto^a,b	Macquarie University	1,097	323
Arizona State University	RMIT University	1,064	290
University of Maryland^b	University of Technology, Sydney	1,056	226

U.S. and University of Toronto data are based on Sanchez (2016, 2017).

University among the top 100 based on the Times Higher Education index (https://www.timeshighereducation.com/world-university-rankings).

The ARC—the largest and most prestigious public research funding body in Australia, which provides the lion’s share of funding for planning research—is notoriously gender biased. While males were the lead investigators on 59 “urban and regional planning” ARC grants between 2010 and 2018, females led only 32 grants. Although differences are small among lecturers, the “funding gender gap” broadens considerably at the professorial levels. Male planning professors hold 6.4 times as many ARC grants as female planning professors, although there are only 2.7 times more male professors than female professors. This gender gap might be due to the fact that currently, in the ARC College of Experts, which evaluates grant applications, only 70 of the 176 members are female (Pojani et al. 2018).

How does Australia compare to major centers of planning scholarship, such as the United States?

To place the Australian situation in perspective, the median citation metrics are compared with those of planning academics in the United States (Sanchez 2016, 2017). Table A-5 in the online supplement shows the top 10 universities in the respective countries in terms of median citations. In Australia, the average level of citations per person is much lower than in the United States. Comparisons are difficult because research funding, workloads (e.g., teaching requirements), and expectations for scholarship differ between the two countries and between planning programs within each country. However, this large difference in the average level of citations per person might be partly explained by the fact that the United States has a population that is 14 times larger than Australia’s and thus has a much larger “domestic market” of planning issues and planning academics who cite each other’s work. Australia’s geographical isolation and lean research funding has tended to preclude high levels of international planning research. Our comparisons contrast with those produced by the ARC in its latest Excellence in Research (ERA) report (Table A-5 in the online supplement).

Bibliometrics: Google Scholar versus Scopus

Table A-6 in the online supplement shows that all the metrics reported by Scopus are much lower than the metrics reported by Google Scholar. The total number of publications reported by Scopus (for all planning schools) is less than half the number reported by Google Scholar. Similarly, for total citations, the number reported by Scopus is less than half the number reported by Google Scholar.

Individually, the most striking example is University of New South Wales, where citation counts differ between Google Scholar and Scopus by around 12,000. Curtin University and the University of Tasmania have a higher citation impact based on Scopus data than leading Go8 universities, despite their small faculties and modest publication numbers. Macquarie University and the University of Queensland have the highest citation counts in Scopus. Notably, at the University of Queensland, nearly 60 percent of the citations captured by Scopus have been contributed by a single academic (who departed from the program after these data were collected).

On the basis of these findings, we conclude that employing Scopus data would be problematic for planning academics who work in large interdisciplinary schools, which combine nonprofessional disciplines. The publications produced by geography, political science, or environmental management academics are more likely to be included in Scopus, thus disadvantaging planning academics because faculty administrators might not recognize the difference between databases. This is a critical issue that can affect opportunities for promotion and tenure.

Discussion and Conclusion

There are four main findings of this benchmarking study: (1) wide gender gaps exist in Australian planning academia; (2) the mean number of publications is 36 per person, or 3 per person per year; (3) the mean number of citations is 527 per person, or 48 per person per year; and (4) planning journal IFs are low. There are some observable institutional differences. In Go8 universities, all metrics are much higher than average. However, not all Go8 universities are among the top 10 in terms of publications and citations.

Productivity is not solely a function of career trajectory. While the number of publications increases as academics progress through the ranks, academic output tends to plateau once individuals reach the top rank of professor. In contrast, citation counts increase steadily as academics progress along their career, nearly doubling from one academic level to the next. However, overall a strong statistical relationship exists between the number of publications and citations. While planning academics publish in a diverse range of journals, the IFs of those journals are generally low—ranging from 0.69 to 2.56. Furthermore, many reputable journals do not have an IF.

What constitutes a productive Australian planning academic is also configured by gender. All metrics (publications, citations, grants) are lower for women than for men. While women are entering the profession in larger numbers than men, planning programs may have emphasized the retention and/or promotion of men, and this has translated into much larger numbers of men in the professorial ranks. Alternatively, this finding may reflect a context in which male academics occupy higher ranks because they are older and more experienced (having been hired in the past, when conditions were different). Also, women are underrepresented on the ARC College of Experts, which evaluates grant applications (Pojani et al. 2018). These patterns indicate that significant institutional barriers may remain, potentially hindering female academics’ chances of tenure and/or promotion. Further research, possibly applying both qualitative and quantitative methods, is warranted (e.g., surveys, interviews, ethnography) to precisely identify institutional or structural barriers. Factors differentially impacting productivity might include family/child-rearing responsibilities, workloads, community engagement, pastoral care responsibilities, and levels of academic mentoring and sponsorship available to junior female academics.

A sizeable body of research that has considered gendered patterns in the Australian academy overall (not only planning) provides pointers.¹¹ Various commentators have suggested that the academy is a hostile work environment for women. The shift in the 1990s from collegial to managerial decision making has entrenched the gendered character of university power relations and contributed to the predominance of women in the lower ranks. Women just beginning or resuming their careers (e.g., after maternity leave) are particularly vulnerable. As a consequence, not enough women remain in higher education. Lacking critical mass, senior female academics are unable to influence management culture, while at the same time early-career female academics end up underprovided with networks, mentoring, and encouragement. Even women who reach senior levels encounter the power of the male hegemony, which is prepared to accommodate some women but not to have its dominance challenged (Asmar 1999; White 2001).

The average level of citations per person is much lower in Australia than in the United States. Disparities in granting success between the two countries may explain this difference, in addition to the much larger size of the profession in the United States. In Australia, the ARC is the main funder of planning research, and between 2006 and 2016, a majority (61 percent) of planning academics did not attract any ARC grants (ARC 2016; Pojani et al. 2018). This situation contrasts with the United States, where most funding for planning comes from sources other than the National Science Foundation (the U.S. equivalent of ARC) and a significantly higher level of financial support is provided to planning (Troy 2013).

These findings have the potential to inform university administrators about the problematic use of some comparative bibliometric data in performance review, tenure, and promotion activities. Our findings also enable comparison between the research productivity of Australian planning academics and those internationally. It must be noted, though, that while we have conducted a practical study based on metrics, we do not advocate that such metrics remain the primary enumerator of planning research productivity. The quality and value of research cannot be measured or fully captured through numbers alone. Recruitment and promotion panels at universities would do well to read academics’ published work—ideally under conditions of anonymity—appraising research quality against diverse criteria, not relying solely on metrics as a shorthand proxy for quality. While difficult and time-consuming, such evaluation practices are necessary to counter the shortcomings of metrics-based assessments.

Supplemental Material

Pojani_online_supp – Supplemental material for Research Productivity of Australian Planning Academics: A Bibliometric Analysis

Supplemental material, Pojani_online_supp for Research Productivity of Australian Planning Academics: A Bibliometric Analysis by Dorina Pojani, Jaime Olvera-Garcia, Neil Sipe and Jason Byrne in Journal of Planning Education and Research

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: A grant was provided by the University of Queensland to fund this research.

Notes

Supplemental Material

Supplemental material for this article is available with the manuscript on the JPER website.

ORCID iD

Jason Byrne

Author Biographies

Dorina Pojani is a senior lecturer in urban planning at the University of Queensland, Australia. She specializes in physical planning (urban design, transport, housing).

Jaime Olvera-Garcia is a recent graduate of the University of Queensland, Australia. His PhD research focused on regional planning and natural resource management.

Neil Sipe is a transportation planner at the University of Queensland, Australia.

Jason Byrne teaches at University of Tasmania, Australia. He researches urban political ecologies of green space, climate change adaptation, and environmental justice.

References

Aguillo

Isidro

. 2012. “Is Google Scholar Useful for Bibliometrics? A Webometric Analysis.” Scientometrics 91 (2): 343–51.

Asmar

Christine

. 1999. “Is There a Gendered Agenda in Academia? The Research Experience of Female and Male PhD Graduates in Australian Universities.” Higher Education 38:255–73.

Australian Research Council. 2016. “ARC Grants.” Accessed January 19, 2016. http://rms.arc.gov.au/RMS/Report/Download/Report/d6b15b2b-3a50-4021-8e6f-6c7ef1cba553/0.

Batterbury

Simon

Byrne.

Jason

2017. “Australia: Reclaiming the Public University.” Social Epistemology Review and Reply Collective. Accessed October 4, 2018. https://social-epistemology.com/2017/07/13/international-responses-to-the-academic-manifesto-reports-from-14-countries-willem-halffman-and-hans-radder/2018.

Bornmann

Lutz

Hans-Dieter

Daniel

. 2008. “What Do Citation Counts Measure? A Review of Studies on Citing Behavior.” Journal of Documentation 64 (1): 45–80.

Bornmann

Lutz

Hans-Dieter

Daniel

. 2009. “The State of h-index Research. Is the h-index the Ideal Way to Measure Research Performance?” EMBO Reports 10 (1): 2–6.

Brembs

Björn

Button

Katherine

Munafò

Marcus

. 2013. “Deep Impact: Unintended Consequences of Journal Rank.” Frontiers in Human Neuroscience. doi:10.3389/fnhum.2013.00291.

Byrne

Jason

. 2017. “Tracing the History of Australian Planning Research and Its Influence on Practice.” In The Routledge Handbook of Australian Urban and Regional Planning, edited by Vella

Sipe

Schooens

, 335–53. New York: Routledge.

Campbell

Philip

. 2008. “Escape from the Impact Factor.” Ethics in Science and Environmental Politics 8:5–7.

10.

Checkoway

Barry

. 1997. “Reinventing the Research University for Public Service.” Journal of Planning Literature 11 (3): 307–19.

11.

Connell

Raewyn

. 2015. “Australian Universities under Neoliberal Management: The Deepening Crisis.” International Higher Education 81:23–25.

12.

Dodd

Tim

. 2017. “Education Exports Are Worth $28 Billion a Year, Nearly 20pc More Than We Thought.” Australian Financial Review. Accessed September 3, 2018. https://www.afr.com/leadership/education-exports-are-worth-28-billion-a-year-nearly-20pc-more-than-we-thought-20171005-gyvc8v.

13.

Frank

Nancy

. 2008. “Measuring Public Service: Assessment and Accountability—To Ourselves and Others.” Journal of Planning Education and Research 27 (4): 499–506.

14.

Harzing

Anne-Wil

. 2010. The Publish or Perish Book: Your Guide to Effective and Responsible Citation Analysis. Melbourne, Australia: Tarma Software Research.

15.

Hicks

2012. “Performance-Based University Research Funding Systems.” Research Policy 41 (2): 251–61.

16.

Hirsch

Jorge

. 2005. “An Index to Quantify an Individual’s Scientific Research Output.” Proceedings of the National Academy of Sciences of the United States of America 102 (46): 16569–72.

17.

Hurt

Christine

Yin

Tung

. 2006. “Blogging while Untenured and Other Extreme Sports.” Washington University Law Review 84 (5): 1235–55.

18.

Hyland

Ken

. 2016. Academic Publishing: Issues and Challenges in the Construction of Knowledge. Oxford: Oxford University Press.

19.

Innovative Research Universities. 2018. “Australian Researchers’ Productivity Boom.” Accessed September 3, 2018. https://www.iru.edu.au/news/australian-researchers-productivity-boom/.

20.

Jöns

Hoyler

. 2013. “Global Geographies of Higher Education: The Perspective of World University Rankings.” Geoforum 46:45–59.

21.

Kousha

Kayvan

Thelwall

Mike

. 2008. “Sources of Google Scholar Citations outside the Science Citation Index: A Comparison between Four Science Disciplines.” Scientometrics 74 (2): 273–94.

22.

Kousha

Kayvan

Thelwall

Mike

Rezaie

Somayeh

. 2010. “Using the Web for Research Evaluation: The Integrated Online Impact Indicator.” Journal of Informetrics 4 (1): 124–35.

23.

Krumholz

Norman

. 1986. “From Planning Practice to Academia.” Journal of Planning Education and Research 6 (1): 60–65.

24.

Pojani

Dorina

Olvera-Garcia

Jaime

Byrne

Jason

Sipe.

Neil

2018. “Australian Cities Are Crying Out for Better Planning, but the Research Funding Is Missing.” The Conversation, February 23, 2018.

25.

Postman

Neil

. 1993. Technopoly: The Surrender of Culture to Technology. New York: Vintage.

26.

Priem

Jason

Taraborelli

Dario

Groth

Paul

Neylon

Cameron

. 2010. “Altmetrics: A Manifesto.” Accessed November 9, 2016. http://altmetrics.org/manifesto.

27.

Riordan

Sarah

. 2011. “Paths to Success in Senior Management.” In Gender, Power and Management: A Cross Cultural Analysis of Higher Education, edited by Bagilhole

White

, 110–39. Basingstoke, UK: Palgrave Macmillan.

28.

Sanchez

Tom

. 2014. “Academic Visibility for Urban Planning and the Webometric Future.” Journal of the World Universities Program 6 (2): 37–52.

29.

Sanchez

Tom

. 2016. “Planning Citation Analysis: Update.” Accessed September 13, 2016. http://tomwsanchez.com/planning-citation-analysis-update/.

30.

Sanchez

Tom

. 2017. “Faculty Performance Evaluation Using Citation Analysis: An Update.” Journal of Planning Education and Research 37 (1): 83–94.

31.

Savonick

Danica

Davidson

Cathy

. 2017. “Gender Bias in Academe: An Annotated Bibliography of Important Recent Studies.” Accessed June 28, 2017. http://blogs.lse.ac.uk/impactofsocialsciences/2016/03/08/gender-bias-in-academe-an-annotated-bibliography/#new.

32.

Simons

Kai

. 2008. “The Misused Impact Factor.” Science 322 (5899): 165.

33.

Spain

Daphne

. 1992. “Creating and Defending Links between Teaching, Research, and Public Service.” Journal of Planning Education and Research 12 (1): 77–79.

34.

Stevens

Garry

. 1990. “An Alliance Confirmed Planning Literature and the Social Sciences.” Journal of the American Planning Association 56 (3): 341–49.

35.

Stiftel

Bruce

Rukmana

Deden

Alam

Bhuiyan

. 2004. “A National Research Council-Style Study.” Journal of Planning Education and Research 24:6–22.

36.

Taylor

Paul

Braddock

Richard

. 2007. “International University Ranking Systems and the Idea of University Excellence.” Journal of Higher Education Policy and Management 29 (3): 245–60.

37.

Thoreau

Henry David

. 1854. Walden, and on the Duty of Civil Disobedience. Boston: Ticknor and Fields.

38.

Troy

Patrick

. 2013. “Australian Urban Research and Planning.” Urban Policy and Research 31 (2):134–49.

39.

Wachs

Martin

. 1994. “The Case for Practitioner Faculty.” Journal of Planning Education and Research 13 (4): 290–96.

40.

White

Kate

. 2001. “Women in the Professoriate in Australia.” International Journal of Organisational Behaviour 3 (2): 64–76.

41.

Wiewel

Wim

Carlson

Virginia

Friedman

Suzanne

. 1996. “Planning the New Urban University: The Role of Planning Departments.” Journal of Planning Education and Research 16 (2): 127–35.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.59 MB