Abstract
The planning literature has taken a recent interest in journal article citation counts, which are often used to measure the scholarly impact of articles, authors, or university departments. However, little is known about the factors that determine citation counts for planning-related articles. We find that citation counts in planning vary across planning topics and are also influenced by other journal, author, and article-related factors. We provide recommendations to planning researchers for increasing the impact of their research, and advise consumers of citation counts in planning to consider making particular adjustments to the counts to make them more meaningful.
citation counts, impact factors, meta-research, planning scholarship, planning journals, multilevel modelingThousands of academic articles are published across thousands of academic journals each year. While many of these articles present new research, a small percentage of them can be categorized as meta-studies of past research. Meta-studies generally consist of either (1) a meta-analysis that synthesizes previous findings from multiple studies on a particular topic to extract an overall message or (2) meta-research that examines various aspects of the research process itself to better understand research standards and practices within or across academic disciplines. The planning literature has shown increasing interest in meta-studies, with several meta-analyses (cf. Berke and Godschalk 2009; Ewing and Cervero 2010; Fang 2015; Park, Huang, and Newman 2016; Stevens 2017), and meta-research articles (cf. Du Toit, Boshoff, and Mariette 2017; Goldstein and Carmin 2007; Goldstein and Maier 2010; Lyles and Stevens 2014; Sanchez 2017; Stevens, Lyles, and Berke 2014) having been published in the past decade or so.
The most recent meta-research in planning has focused on citations, whether measured at the level of articles, journals, authors, or university departments. Citations are an important concern for planning scholarship in part because of how they are used. On one hand, planning scholars are expected to be productive researchers and to make significant contributions to knowledge; on the other hand, it is not easy to measure the impact that a planning article or researcher has had on planning theory and practice. In the absence of direct measures of scholarly productivity and impact, citations have been commonly used instead as a proxy measure (Bornmann et al. 2012). Based on a general belief that higher citation counts indicate greater impact, various decision-makers use citation counts to help determine career advancement for academic researchers, to evaluate the collective impact of universities and departments, and to guide research grant funding allocations (Antoniou et al. 2015; Bornmann and Daniel 2007; Knobloch-Westerwick and Glynn 2013; Stremersch, Verniers, and Verhoef 2007).
Although there is a large body of literature on citations in other fields, researchers have only recently begun to explore citations in the planning literature to better understand citation behavior and how citations vary across planning articles, researchers, and university programs. Two recent meta-research articles in planning (Sanchez 2017; Du Toit, Boshoff, and Mariette 2017) used descriptive methods, in part, to explore whether citations vary across such factors as academic faculty rank and methodological approach. These studies represent a useful first step in the direction of understanding citation behavior in planning, but they are limited in the sense that they examine only a small number of factors and their methods do not control for potential confounds.
In this article, we build on these studies as well as past research from fields outside of planning to develop and test a model aimed at answering the following question about planning scholarship: why do some articles in planning-related journals get cited more than others? Our model includes factors that have been discussed in planning meta-research (such as the subject matter of the article and the scholarly reputation of the journal), as well as factors that have been identified by other disciplines as significant predictors of article citation counts (such as the journal impact factor and the number of co-authors). We begin by reviewing four recent and relevant meta-research articles in planning and summarizing findings from previous studies on factors that help to determine journal article citation counts. We then describe our data collection and analysis methods. Next, we present and discuss our findings and then conclude with thoughts on the use of citation counts and meta-research in planning scholarship.
Meta-Research on Planning Scholarship and Citations
Goldstein and Carmin (2007) conducted a meta-study of articles published in Journal of the American Planning Association from 1963 to 2002 to examine trends in how planning has developed as an academic discipline. They observe that planning is very diverse with “widespread and liberal borrowing of ideas, concepts, and tools from other disciplines” (p. 69). Some of this diversity is reflected in the wide range of planning specializations they found to be represented across their sample of articles, including planning theory and history, practice/research methods, land use and environment, housing and community development, economic development, transportation and infrastructure, and urban design, among others. They found that some of these specializations were represented more frequently than others, which perhaps suggests that some topics in planning are more popular than others. If so, then it is possible that papers written about relatively popular topics might receive more citations on average than papers written about less popular topics.
Goldstein and Maier (2010) conducted a survey of planning scholars regarding the value they place on various planning-related journals. They used the survey results to rank the “35 most important journals in planning scholarship by peer evaluation,” and they found that two journals (Journal of the American Planning Association and Journal of Planning Education and Research) had by far the strongest reputations among planning scholars. They also acknowledge that journal impact factors represent an alternative way to measure a journal’s reputation or importance. When they compared impact factors with the reputational rankings derived from the survey results they found no correlation between the two, suggesting that impact factors and the survey-based rankings are generally nonoverlapping measures of a journal’s importance. To the extent that an article’s citation count depends on which journal published the article, it is possible that both the impact factors and the reputational rankings of planning journals might have an influence on the number of citations their papers receive.
Sanchez (2017) conducted a citation analysis of planning scholarship. He compiled a list of all faculty members from Association of Collegiate Schools of Planning (ACSP) member schools, and then utilized Google Scholar data to calculate total citation counts for university planning faculty and programs. 1 While his descriptive analysis focuses primarily on how faculty citation counts vary across programs, he also shows that the citation counts are clearly related to faculty rank, with Full Professors having many more citations on average than Associate or Assistant Professors. However, he does not explore factors that help to determine citation counts at the level of an individual article. It might be the case, for example, that senior scholars have more citations than junior scholars simply because senior scholars have more publications, which does not necessarily mean that a given paper published by a senior scholar would receive more citations than it would have received had it been published by a junior scholar instead. In light of such ambiguity, Sanchez (p. 83) encourages planning researchers to “undertake more analysis of planning scholarship to understand scholarly performance and impact.”
Most recently, Du Toit, Boshoff, and Mariette (2017) reviewed the methodologies used in papers published by Journal of Planning Education and Research from 1996 to 2005, focusing in part on whether the papers featured quantitative, qualitative, or mixed (i.e., both quantitative and qualitative) methods. They found that more than one-half of the papers they studied employed just qualitative methods, compared with roughly one-third that employed just quantitative methods. Despite this apparent preference for qualitative methods in at least one planning journal, however, they found that papers that used quantitative methods received more citations on average than papers that used qualitative or mixed methods. Yet their analysis was limited, as it was strictly bivariate and did not control for potential confounding factors. It is possible, for example, that quantitative papers differ from qualitative papers in some other meaningful ways that might be responsible for the difference in citations that the two kinds of papers receive.
Research on Factors that Help to Determine Citation Counts
Whereas the planning literature has only recently begun to explore factors that help to determine citation counts for planning-related articles, there is a well-developed body of literature on determinants of citation counts outside of planning. Researchers have studied a wide range of potential determinants, from those with high face value (such as journal impact factor) to seemingly extraneous factors (such as the use of punctuation marks in article titles). In a recent review of 198 studies that identified factors that help to determine article citation counts, Tahamtan, Afshar, and Ahamdzadeh (2016) grouped the factors into three categories: Journal-related factors, Author-related factors, and Paper-related factors. We now provide a condensed summary of their review, focusing on factors that have been found to be statistically significant predictors of citation counts.
Journal-Related Factors
Impact factor: Journals are distinguished, in part, by their impact factors, which give a general sense of how often articles in the journals have been cited. Many studies 2 have found that citation counts tend to increase as the impact factor of the journal that published the article increases. Although this relationship is technically endogenous (in the sense that citation counts influence impact factors and vice versa), it is a convention in the literature to focus on the influence that impact factor has on the number of citations an article receives.
Author-Related Factors
Sex of author: There is no obvious reason to think that males will write better articles than females. Nevertheless, article citation counts tend to be higher for male lead/sole authors than for female (Knobloch-Westerwick and Glynn 2013; Maliniak, Powers, and Walter 2013). This is possibly an example of the so-called “Matilda Effect” (Rossiter 1993), under which female researchers receive less credit than male researchers for their respective scholarly contributions.
Author’s previous citations: When it comes to citations, familiarity tends to breed more citations. In general, citation counts tend to increase with the number of previous citations the article’s lead (or sole) author has received on other publications (Walters 2006; Tang, Wang, and Kishore 2014; Yu et al. 2014). This has been labeled the “Matthew Effect,” under which the “rich get richer” (i.e., scholars with many citations get even more), while the “poor get poorer” (i.e., scholars with few citations do not get many more; Merton 1968).
Paper-Related Factors
Age of paper: The age of an article appears to have a positive, nonlinear influence on the article’s citation count. An article’s citation count tends to rise relatively quickly in the first few years after publication. 3 As time passes, however, the information in the article can become increasingly outdated and obsolete (Barnett and Fink 2008), such that the number of additional citations the article receives tends to decrease with each additional year (Meadows 2004; Sin 2011; Wang, Yu, and Yu 2009).
Topic of paper: Citations often vary across different topics and subfields in a discipline, 4 with some topics/subfields being more popular than others. It also tends to be true that papers from smaller disciplines receive fewer citations than papers from larger fields (Bornmann et al. 2012).
Length of paper: Many journals set limits on the permissible length of articles, which might inadvertently reduce the number of citations the articles ultimately receive. A large number of studies indicate that citation counts tend to increase with the length of the article, 5 possibly because longer articles have more material that can be cited and/or because longer articles might have addressed the paper’s topic more thoroughly than shorter articles.
Number of authors: Citation counts tend to increase with the number of co-authors. 6 In general, more co-authors is likely to mean more visibility for the paper, in the sense that each co-author is likely to have their own set of colleagues and followers who does not completely overlap with that of other co-authors. More co-authors also mean more potential for the paper to receive self-citations, wherein co-authors cite their own paper in subsequent papers. It is also possible that co-authored papers are of higher quality than sole-authored papers, simply because more authors can mean more capacity and expertise at hand for writing good papers.
Type of paper and research design: Some types of papers are cited more than others. In particular, papers that provide a review of many previous papers on a given topic tend to be cited more than papers that present original research. 7 Likewise, papers that utilize certain research designs are cited more than others. For example, systematic reviews and meta-analyses are cited more than observational studies. 8 There is also some evidence from the planning literature that papers with quantitative research designs receive more citations than papers with qualitative or mixed methods designs (Du Toit, Boshoff, and Mariette 2017).
Characteristics of the title, abstract, and keywords: The decisions that authors make regarding features such as titles and keywords might have implications for the subsequent citation counts of their articles. First, one study found that articles with titles that are phrased in the form of a question tend to be cited less often than articles with titles phrased as declarative or descriptive statements (Jamali and Nikzad 2011). Second, articles with shorter titles tend to receive more citations than articles with longer titles. 9 Third, papers with punctuation marks (e.g., hyphens, commas, colons, or brackets) in the titles might be cited more than papers with only letters in their titles (Buter and van Raan 2011). Fourth, papers tend to receive more citations when their title reports the study design than when it does not (Antoniou et al. 2015). Last, citation counts tend to increase with the length of the abstract (Annalingam et al. 2014; Falagas et al. 2013; Rostami, Mohammadpoorasl, and Hajizadeh 2014), and/or with the number of keywords (Rostami, Mohammadpoorasl, and Hajizadeh 2014; So et al. 2015).
Use of figures and/or appendix: Citation counts tend to increase with the number of figures used to present results (Ayres and Vars 2000; Miettunen and Nieminen 2003) and the inclusion of appendices (Stremersch et al. 2015; Stremersch, Verniers, and Verhoef 2007).
Characteristics of references: Citation counts appear to depend, in part, upon the number of other works that were cited in the article. Many studies have detected a positive relationship between the number of references included in an article and the article’s citation count. 10
Accessibility: Increasing the accessibility of a paper can help to increase its citation count (Henneken et al. 2006; Rees, Ayling-Rouse, and Smith 2012; Yue and Wilson 2004). One effective way to increase a paper’s accessibility is through self-archiving, whereby the paper’s author makes an electronic copy of the paper freely available online.
Data Collection, Variables, and Modeling
Research Contribution
This article contributes to the growing body of meta-research on planning scholarship and to the established body of literature on factors that help to determine article citation counts. As we describe in more detail below, we build on each of the four afore-mentioned planning meta-research articles in one or more ways, and we examine whether factors that were identified by previous studies to be predictors of citation counts in other literatures are also predictors of citation counts in the planning literature. By doing so, we provide insights into why it is that some planning-related articles get cited more than others.
Data Collection
We created a database of articles published in planning-related journals along with variables that measure characteristics of the articles, their authors, and the journals that published the articles. We included variables to measure characteristics that have been found in previous studies to help determine citation counts. We created the database in the summer of 2017. 11
To focus on articles in planning-related journals, we began by limiting our sample to articles published by faculty members from ACSP member schools. We used the list of all faculty from ACSP member schools that Sanchez (2017) compiled, who also provided us with a copy of the list. The list was current as of 2016. Due to resource constraints we further limited our sample to faculty with Google Scholar 12 profiles, which made it much easier to find all of the faculty members’ publications. We then identified articles by those faculty member authors 13 that were published between 2011 and 2016 in either the thirty-five “most important” planning-related journals as identified in the study conducted by Goldstein and Maier (2010), or the remaining additional thirty-three planning-related journals from the “urban studies” and “planning & development” categories of the InCites Journal Citation Reports. 14 The InCites dataset did not include five of the thirty-five journals that Goldstein and Maier identified, so we excluded those five journals from our analysis. 15 There were 1,100 faculty on the ACSP member list as of 2016, with 392 having a Google Scholar profile. Out of the 392 authors, 241 had at least one article published between 2011 and 2016 in our list of sixty-three planning-related journals. At this stage, our dataset contained 241 authors and sixty-three journals, with a total of 652 articles. 16 We then chose to drop from our sample the seventy-two papers for which the paper’s citation count represented more than 10 percent of the author’s total citations. 17 Our final dataset contained 222 authors and sixty-two journals, with a total of 580 articles.
Table 1 shows the sixty-two journals and the number of articles from each journal that are included in our sample. The table also shows each journal’s 5-year impact factor and the reputational rating that we adapted from Goldstein and Maier (2010). At the time of their study, the top 5 journals by reputation were Journal of the American Planning Association, Journal of Planning Education and Research, Urban Studies, Housing Policy Debate, and Journal of Urban Affairs. Our set of sixty-two journals ranges from “flagship” journals in planning (such as Journal of the American Planning Association and Journal of Planning Education and Research) to journals from related disciplines (such as Urban Geography and Journal of Urban Economics). The journal impact factors range from a low of 0.10 to a high of 5.02, with a mean of 1.96 and a median of 1.85. The most frequently occurring journal in our sample was Transportation Research Record, with sixty-nine articles. Interestingly enough, this journal has both a low impact factor and a low reputation score. Four more journals (including Journal of Planning Education and Research and Journal of the American Planning Association) had thirty-six or more articles, with all remaining journals having twenty or fewer. Ten of the journals had just one article each, including the journal with the lowest impact factor (Open House International).
Sixty-Two Urban Planning Journals Included in This Study.
“Reputation” measures the number of times that a journal was included by planning scholars in their “top 5” most important planning journals in the survey conducted by Goldstein and Maier (2010), such that higher numbers indicate a stronger reputation. Journals without a reputation score were not ranked in the top 35 most important planning journals overall.
Variables
As shown in Table 2, we measured thirty-nine variables in total: one variable for article citation counts, two journal-related variables, two author-related variables, and thirty-four paper-related variables. To measure many of the variables (such as the number of authors, the length of titles, and so on), we collected and reviewed each of the articles in our database. Variables that measure author characteristics are based on characteristics of the lead/sole author, which is a convention in the citation literature. Our dependent variable is the article citation count, which measures the number of times the article has been cited in other journal articles as well as in other types of publications, such as books, book chapters, reports, dissertations, and conference papers. Our journal-related variables measure journal impact factors and the reputational rankings from Goldstein and Maier (2010), 18 and our author-related variables measure whether the lead/sole author is female or male and each lead/sole author’s total citation count. We measured the sex of the author as a binary variable with 1 = female and 0 = male, and we performed a natural log transformation of the author citation count variable to improve the normality of its distribution.
Variables Included in This Study.
Note: Authors refers to the authors of this paper.
We measured the age of each paper with a set of binary variables that indicate the paper’s publication date, expressed as the difference in years between the publication date and 2017. We used binary variables because we expected the relationship between article citation count and the age of the article to be nonlinear. We use the one-year variable (which corresponds to a publication date of 2016) as the reference category in our regression models.
To measure paper topic, we assigned each paper to one of fifteen topic categories, using the same broad set of fifteen categories (or “tracks”) that is used each year by ACSP to group papers for its annual conference. We looked at paper titles, abstracts, and keywords to determine each paper’s topic. In cases where a paper might reasonably be assigned to two or more different topic categories, we exercised our best judgment to select what seemed like the most appropriate single category. After assigning the papers to categories we found that three categories (Planning History, Regional Planning, and Planning Education) contained fewer than ten papers each. We then combined Planning History with Planning Theory and combined Regional Planning with Land Use Policy and Governance, because we felt that these were appropriate combinations based on subject matter. We kept Planning Education as its own category, as there was no other appropriate category to combine it with. We use the Transportation category as the reference category in our regression models, because it was the category that contained the largest number of papers.
We also examine whether citation counts depend on paper accessibility. It stands to reason that the easier it is to access a paper, the more likely it is that other researchers will read and cite it. With this in mind, we included a Free Access variable in our model that is equal to 1 if a full-text electronic version of the paper was freely accessible online, such as through Google Scholar, ResearchGate, Academia.edu, or the author’s personal or institutional webpage. Measurement for the remaining variables is self-explanatory.
Modeling Citation Data
Citation data display certain features that complicate efforts to model them. One of these features is that citation data display a nested (or multilevel) structure, in the sense that papers are “nested” within authors and within journals. By default, regression models are typically set up based on an assumption that the observations in the dataset are independent of each other. This assumption is likely to be violated, however, when two or more papers by the same author are included in a citation data study and/or when two or more papers from the same journal are included. For example, if some authors or journals are more well-known and respected than others, there might be an “author effect” or a “journal effect” that causes their papers to receive more citations on average than papers written by different authors or published in different journals, other things being equal.
The current best statistical approach to modeling nested data is multilevel modeling (MLM), also called hierarchical modeling. While MLM has been around for at least a few decades, planning researchers have only just begun to use it (Ewing et al. 2015; Tian et al. 2015). MLM accounts for dependence among observations, in this case the dependence of articles on the characteristics of authors and journals. If this dependence is not accounted for in the model, the standard errors of the regression coefficients will typically be underestimated and the coefficient estimates will be inefficient. MLM overcomes these limitations, accounting for the dependence among observations and producing more accurate coefficient and standard error estimates (Raudenbush and Byrk 2002).
When an outcome varies systematically in two dimensions (such as citation counts that vary with authors and journals) and random effects are present, the resulting data structure is best represented by a cross-classified random effects model (see Raudenbush and Byrk 2002, ch. 12). A cross-classified random effects model is a special class of multilevel model in which lower level units are nested within two or more higher level units. The two higher levels in this study are journals and authors. A cross-classified random effects model allows us to study the effects of journal differences while controlling for author effects, and the effects of author differences while controlling for journal effects. (See the online appendix for more details on our modeling approach).
To our knowledge, this is only the second application of a cross-classified random effects model to the planning field. Ewing et al. (2005) used this method to model ratings of main street scenes in terms of scene characteristics and viewer characteristics. The model indicated what qualities of main streets distinguished them from other highways.
Limitations
Our analysis has some important limitations that we wish to acknowledge.
Our sample of planning-related journals: Our intention is to study citation counts for papers published in “planning-related journals,” yet the definition of what counts as a planning-related journal is ultimately subjective and there is no authoritative list of planning-related journals. As a result, we had to exercise discretion in deciding which journals to include in our study. We chose to include journals that were identified in the survey conducted by Goldstein and Maier (2010) and/or that were included in the “urban studies” or “planning & development” categories from the InCites database. Although the reader might be of the opinion that our list either includes some journals that are not planning related or excludes some journals that are planning related, it is unclear whether/how the inclusion or exclusion of any given journal would affect our findings regarding which factors influence citation counts.
Our sample of authors: Resource limitations precluded us from including in our study every single paper that was published in our sample of journals from the years 2011–2016. We limited our sample of papers in two ways: (1) by including only those papers that were authored by faculty that were employed by ACSP member schools and that had a Google Scholar profile (profile) and (2) by including only those papers with citation counts that represented less than 10 percent of the author’s total citation count. It is unclear whether or how excluding papers written by researchers without profiles has affected our findings. On one hand, it is probably true that researchers who choose to create a profile for themselves differ in some systematic way from researchers who do not create a profile; on the other hand, it is not immediately obvious that any such differences would necessarily result in a different pattern of factors that influence article citation counts for authors with or without profiles. Future research can help to resolve this uncertainty by studying factors that influence article citation counts for researchers who do not have profiles. With respect to our second sample limitation, we consider it likely that papers with citation counts that represent more than 10 percent of the author’s total citation count were mostly written by junior scholars who had few previous publications. As a result of dropping these papers, our sample is probably weighted more heavily in the direction of established scholars. However, it is again unclear whether the factors that determine citation counts for established scholars differ from those that determine citation counts for junior scholars. This is another question for future research to explore.
Our variables: Resource constraints limited the scope of the variables that we were able to measure and include in our study. Although we did not measure every plausible factor that might have an effect on citation counts, we made a point of including multiple variables from each of the three categories of factors (i.e., journal, author, and paper-related characteristics) that have been found in previous studies to influence citation counts. We also included a new variable in the form of the journal reputation score adapted from Goldstein and Maier (2010), though it is worth noting that journal reputations among planning scholars might have changed since the time of that study.
Journal impact factors and citation counts: The citation literature commonly treats citation count as the dependent variable and journal impact factor as the independent variable. In other words, the literature treats impact factors as a partial cause of citation counts. In reality, however, we know that the reverse is also true: impact factors depend, in part, on citation counts, as any given journal’s impact factor is determined by how many times the journal’s papers have been cited. Our decision to treat impact factor as the cause rather than the result of citation counts is consistent with previous research and also makes intuitive sense to us. Impact factors are determined by the citations for many papers rather than just one, which means, for example, that impact factors for the journals included in our sample might be more heavily influenced by citation counts from other papers besides those in our study. Meanwhile, any one paper has only one impact factor that influences its citation count. In our view, these considerations suggest that the effect that impact factor has on citation count is likely to be stronger and less confounded than the effect that citation count has on impact factor. We cannot know for sure whether a given paper’s citation count will increase when impact factor increases, which is what we are testing in this study. All of that being said, it is nevertheless a limitation of our study that we do not account for the effect that citation counts have on impact factors.
Citation counts as a measure of impact: Planning research is intended, in part, to provide planning practitioners with knowledge to help them “plan” better. Yet it is difficult to measure the impact that planning research has on planning practice, largely because planners do not necessarily report whether they are using research in their practice. While it is not unheard of for planners to cite research in planning documents, it seems likely that many instances of planners relying on knowledge from research will go uncited, which means that citation counts are unlikely to capture at least some of the influence the research is having on planning practice. Assuming that researchers are doing most of the citing, citation counts arguably do a better job of capturing whether and to what extent other researchers (as opposed to practitioners) are paying attention to and using the research. While we believe that citation counts do reflect scholarly impact at least to some extent, they are unlikely to be a good measure of the impact that research is having on practice. Future research might be able to contribute to our understanding of the impact that planning research has on practice by surveying planning practitioners to find out which articles (if any) they actually read and use to guide their practice.
Other measures of research impact: While citation counts have been the most commonly used measure of scholarly impact, complementary measures are being developed to provide a broader understanding of an article’s reach and influence beyond whether or not it is cited in a journal or other traditional type of publication. For example, so-called altmetrics measure how often articles are discussed on social media, blogs, mainstream news networks, and so on. When an article is discussed in one of these alternative venues that is taken as evidence that the article is reaching a broader audience than the relatively narrow group of people who read academic journals. Studies have found that while altmetrics and citation counts are positively correlated, the correlations tend to be weak (Costas, Zahedi, and Wouters 2015; Thewall et al. 2013), which suggests that almetrics and citation counts partially overlap but also likely measure different dimensions of research impact. Future research can contribute to our understanding of scholarly impact in planning by examining whether and how planning research is disseminated and consumed beyond traditional publication outlets, such as by studying altmetrics and other alternative measures of the attention an article has received and the “buzz” it has created.
Findings from Our Analysis of Citation Counts in Planning-Related Journals
In this section we present our findings, starting with a review of descriptive statistics and then moving to the results of our regression models. Figure 1 shows the distribution of our dependent variable, that is, the number of citations each article in our sample has received. As is common for citation data the distribution is positively skewed, with most of the articles receiving a small number of citations and a few articles receiving a large number of citations. The mean number of citations (17.9) is greater than the median (10), which is greater than the mode (thirty-nine articles had received one citation each). Three-fourths of the articles received twenty-one or fewer citations. The most-cited article had received 440 citations, which is roughly 14 standard deviations higher than the mean count. Seventeen articles had not been cited at all, with thirteen of the seventeen having been published in 2016 and two having been published in 2015.
Frequency of article citations.
Table 3 shows descriptive statistics for our variables, including correlations between article citation count and each of our independent variables. Our sample contains 580 articles, 196 (33.8%) of which had a female lead/sole author. The mean number of total citations for the authors was roughly 2,505, and the total citations for authors ranged from twenty-eight to 56,999. The articles were spread relatively evenly across the publication years (2011–2016), with the fewest (14.6%) published in 2015 and the most (20.0%) published in 2011.
Descriptive Statistics.
This column shows the respective correlation between the number of citations an article has received and each of the independent variables listed in the first column of the table. We use the natural log of the number of citations variable, to improve the normality of its distribution. Numbers in the parentheses are p values associated with the correlation.
More than one-fourth of the articles (27%) were in the Transportation topic category, 18 percent were in Economics, and 14 percent were in Housing. The least-represented categories were Urban Design (2%) and Education (1%). Four of the top 10 most-cited papers were in the Transportation category, two were in Health/Safety, two were in Theory/History, one was in Economics, and one was in Housing.
More than two-thirds of the articles had either one author (33.6%) or two (33.6%). Twenty four of the papers (4.1%) were literature review papers, and 361 (62.2%) used primarily quantitative methods. On average, the papers included fifty-five references, with a median of 48.5. The number of references ranged from three to 246. Nearly one-half (48.3%) of the papers were freely accessible online for download.
The correlations are of no more than moderate size, with the largest correlation (.24) being for journal impact factor. The journal with the highest impact factor (i.e., Landscape and Urban Planning) published two of the ten most-cited papers, and eight of the ten most-cited papers were published in journals with impact factors higher than the mean impact factor of 1.96. The author citation count and number of references each have a correlation of .20, followed by the number of pages at .16, the Theory/History topic variable at .13, and the number of keywords at .10. All other correlations are 0.09 or smaller in magnitude. While the correlation for the female variable is not statistically significant, it is worth noting that the most-cited papers were generally written by males. The highest cited paper was written by a female, but each of the next nine highest cited papers and seventeen of the twenty highest cited papers overall were written by males. The mean number of citations for papers written by males was 18.5 compared with 16.9 for females, though this difference is not statistically significant. 19
We estimated two different regression models: model 1, which includes the journal reputation variable and is therefore based only on the papers from our sample that were published in the twenty-nine journals with reputation scores, and model 2, which is based on the entire sample of papers and therefore does not include the journal reputation variable. The primary value of model 1 is that it enables us to examine whether citation counts vary with journal reputation; however, as it is based on just a subset of the articles in our sample, we focus our discussion of findings primarily on those from model 2 because it is based on our full sample of journals, authors, and papers.
Table 4 shows our model results. Beginning with journal-related factors, the results for model 1 show that the journal reputation score appears to have no influence on article citation counts. With that being established we now focus on the results for model 2. The results suggest that journal impact factor has a positive effect on citation counts, with each 1-point increase in impact factor producing a 27 percent increase in citation counts on average. This means, for example, that a paper published in Journal of Environmental Planning and Management (which has an impact factor of 1.87) would be expected to receive 27 percent more citations than it would have received had it been published instead in Transportation Research Record (which has an impact factor of 0.87), other things being equal.
Results of the Cross-Classified Random Effects Models.
Coefficient is the raw, unexponentiated coefficient from the regression model.
IRR is the incidence rate ratio (or exponentiated coefficient). An IRR is generally considered to be easier to interpret than the unexponentiated coefficient. An IRR of 1.50, for example, can be interpreted to mean that a one-unit increase in the associated independent variable increases the rate of the dependent variable count by 50%. Conversely, an IRR of 0.50 means that a one-unit increase in the associated independent variable decreases the rate of the dependent variable count by 50%. IRRs greater than 1.00 indicate a positive relationship between the independent variable and the dependent variable; IRRs between zero and 1.00 indicate a negative relationship; IRRs equal to 1.00 indicate no relationship.
Results for this variable are in reference to the Topic: Transportation category, which is the reference category and is omitted from the regression model.
Moving to author-related factors, we found no significant difference in citation counts for articles written by females versus males, suggesting that papers in planning journals that are written by females are cited just as often as papers written by males, other things being equal. As we noted above, the most-cited paper in our dataset was written by a female and its citation count was much higher than the mean citation count. The observation for this paper in our dataset also had a very large Cook’s distance value, which is an indication that this observation might be an influential case in our regression model. However, when we estimated a separate regression model that excluded this observation we found that the model results did not change very much. Most importantly, there was still no significant difference in citation counts for female versus male authors when the most-cited paper was excluded from the model.
Our second author-related factor was author citation count. Whereas journal reputation appears to have no effect on citation counts, the reputation of the author does appear to have a positive effect, though the size of the effect is small. Our results show that a one-unit increase in the log of an author’s total citation count increases the citation count of his or her current paper by a modest 0.003 percent.
The remainder of our results is for paper-related factors. As expected, citation counts increase in a nonlinear fashion with years since publication. The largest increases occur in between years 4 and 5, and 3 and 4, and the smallest occurs between years 5 and 6. Citation counts increase on average by 45 percent from year 1 to 2, then by 42 percent from year 2 to 3, 50 percent from year 3 to 4, 53 percent from year 4 to 5, and 6 percent from year 5 to 6.
Our results show that, as expected, article citation counts vary across different planning-related topics. The topic that receives the most citations is Planning Theory/History, which receives 40 percent more citations on average than Transportation, which is the reference category in our models. All other topics receive fewer citations than Transportation, though in some cases the difference is not statistically significant. Focusing on just the significant differences, the topic that receives the fewest citations is Education, at 59 percent fewer citations than Transportation. Next is Diversity at 55 percent, followed by Environmental at 47 percent, Land Use at 34 percent, Analytics at 32 percent, and Economics at 18 percent fewer citations than Transportation. While remaining differences across topics are not statistically significant, some of them are still noteworthy in size, such as for Urban Design (32% fewer), International (22%), Health/Safety (16%), and Planning Process (16%). It is possible that these relatively large differences are not found to be statistically significant because of sample sizes, in that each of these four categories makes up only between 2 and 7 percent of our sample. It might be the case that our sample contains too few papers from these categories to provide enough statistical power for our model to find significant differences between the citation counts for those categories and counts for Transportation.
Figure 2 provides a visual representation of the differences in citation rates for different planning topics, once the other variables in our models have been controlled for. The values in the figure are scaled such that Transportation (the reference category) arbitrarily receives 100 citations, and the values for all other topics represent a proportion of 100. For example, if a Transportation paper and an Education paper had the same scores on all of our independent variables except for the paper topic variable, and if the Transportation paper were to receive 100 citations, then our model would predict that the Education paper would receive forty-one citations.
Citation rates for different planning specializations.
Four other paper-related factors appear to be related to article citation counts. Citation counts increase by 3 percent on average with each additional keyword included with the article. Longer papers tend to receive more citations than shorter papers, with each one-page increase in paper length leading to a 2 percent increase in article citation count. Longer abstracts also seem to matter, with each additional word included in the abstract being associated with a less than 1 percent increase in citation count. Last, each additional reference included in a paper increases the citation count of the paper by less than 1 percent.
This finding for references warrants a bit more exploration. We reported in Table 3 that both the references variable and the literature review variable were positively correlated with article citation counts, but our model results in Table 4 show that only the references variable is statistically significant in our models once other factors are controlled for. This was a surprising finding to us because we expected literature review papers to attract high citation counts, given that past studies have found review papers to receive more citations. We performed some additional analyses in hopes of explaining this unexpected finding, and found evidence of a possible indirect relationship between literature review papers and citation counts that is mediated by references. We speculated that review papers would contain more references than non-review papers, and indeed we found in our sample that review papers contained 112 references on average compared with fifty three for non-review papers, and that the correlation between literature review papers and references was .39. We then reestimated our models in Table 4 while including the literature review paper variable but excluding the references variable, and found that the literature review paper variable was statistically significant; when we added the references variable to the model, it was significant but the literature review variable was not. Taken together, we interpret these findings to mean that whether or not a paper is a literature review paper helps to determine how many references the paper will contain, and the number of references the paper contains helps to determine the number of citations the paper will receive, but being a literature review paper has no direct influence on citation counts once the number of references is controlled for.
Our models included several variables that had no significant influence on article citation counts. Four of these variables were related to the title of the article, including the length, inclusion of punctuation marks, phrasing in the form of a question, and reporting the study’s design. Other nonsignificant variables included the number of authors, the number of figures/tables, the inclusion of an appendix, and whether the article was freely accessible online. We also found no significant difference in citation counts for articles that used quantitative versus qualitative or mixed methods.
So Why Do Some Articles in Planning-Related Journals Get Cited More than Others?
In this article, we explored the question of why some articles in planning-related journals get cited more than others. Our findings suggest that planning-related articles receive more citations when they are published in journals with high impact factors, published by authors with many previous citations, written about planning theory/history or transportation topics, and are long papers with long abstracts, many keywords, and many references. Conversely, citation counts for planning-related articles can be expected to be particularly low when the articles are published in journals with low impact factors and when they are written about topics such as Education, Diversity, Environmental Planning, Land Use, Urban Design, or Analytics.
Our findings are consistent with those from a large number of previous studies that found a positive relationship between journal impact factor and article citation counts. Citation counts in planning-related journals tend to increase relatively rapidly with impact factors, with each 1-point increase in impact factor corresponding to a 27 percent increase in citation count. Assuming that journals with higher impact factors are more widely read than journals with lower impact factors, publishing in a high impact factor journal is likely to extend an article’s visibility to a larger audience, and might also send an implicit signal that the article is of higher quality than articles published in journals with lower impact factors. The very fact that a journal’s impact factor is high signifies that (for whatever reason) the journal has enjoyed success in terms of its articles being cited.
Even though our journal reputation variable specifically measures the self-reported importance of planning journals for planning researchers, that level of importance does not appear to carry over to citation counts. Our results suggest that citation counts do not vary with journal reputation, and that a given paper would be expected to receive the same number of citations if it were published in a low reputation journal that it would have received if it had been published in a high reputation journal. It is worth noting, however, that our reputation variable is based on survey results compiled by Goldstein and Maier (2010) in 2007, more than one decade ago at the time of this writing. It is possible that the reputations of planning journals have changed since that time and that citation counts might actually be found to vary with more contemporary reputational rankings.
The effects of author-related characteristics on citation counts in planning appear to be relatively minor, which is arguably an encouraging finding. In theory, it is arguably better for papers to be evaluated (and cited) based on their own merits, rather than on the merits of their authors. While our results show that article citation counts in planning tend to increase with the number of previous citations the author had received, the size of the effect appears to be quite small, and being female versus male appears to have no effect on citation counts whatsoever. Although previous studies in other fields found that articles written by females tend to receive fewer citations (Knobloch-Westerwick and Glynn 2013; Maliniak, Powers, and Walter 2013), our findings seem to suggest that there is perhaps more equality in the planning literature in the sense that researchers appear to value publications produced by females just as much as those produced by males, at least within the context of citation counts. That being said, the fact that just one-third of the papers in our sample were written by females might be interpreted by some to mean that there is a lack of equality in university planning programs and in the process of publishing journal articles in the first place.
Previous studies have found that some paper topics attract more attention and citations than others, and our findings suggest that this is true in planning scholarship as well. Papers written on Theory/History topics receive the most citations and papers written on Education receive the fewest, other things being equal. Our model results suggest, for example, that if two papers were alike in all other ways that we accounted for (including journal and author-related factors), with one paper being written on a Theory/History topic and the other on an Education topic, the Theory paper would be expected to receive more than twice as many citations than the Education paper. One plausible explanation for the popularity of Theory papers is that they might be of general interest to planning researchers across specializations, such that a broad spectrum of researchers might be interested in planning theory regardless of whether they study transportation or housing or diversity in planning. The topic with the second highest citation rate is Transportation. Although Transportation papers are not necessarily of higher quality or greater importance than papers written about other topics, they do have the likely advantage of attracting more attention than many other planning-related topics from researchers in disciplines outside of the planning field, including large disciplines such as engineering, medicine, and economics. By connecting to large disciplines with many researchers, Transportation papers might receive more citations, in part, because they are likely to be more widely read, and read by researchers who study (and publish on) related issues.
Our findings for article length and the number of references are also consistent with findings from previous studies, and they highlight the potential value of being thorough in addressing the topic of study. First, longer papers receive more citations than shorter papers, which could be a result of the fact that longer papers might have more material to cite than shorter papers. Second, papers with more references receive more citations than papers with fewer references. There are several plausible explanations for this relationship. Each reference that a paper includes helps to make the paper itself more visible, such as through citation-based searching in databases like Google Scholar and Web of Science (Didegah and Thelwall 2013, 1056). Papers with more references are also likely to come from larger fields or specializations, which means that they are likely to have more potential citers. According to the so-called “tit for tat” hypothesis, researchers also tend to cite the works of their ex-citers, which means that each reference included in a paper increases the likelihood that the authors of that reference will see and cite the paper (Webster, Jonason, and Schember 2009). Combined with our finding that literature review papers do not receive more citations than non-review papers, this suggests that a non-review paper with many references would be expected to receive more citations than a review paper with fewer references.
Our findings indicate that the planning literature differs from other literatures with respect to the influence of several paper-related factors on citation counts. We found that adding co-authors to a planning paper does not result in more citations, despite the fact that more authors should mean more exposure. Given that articles from disciplines such as medicine and the natural sciences tend to have many more co-authors than articles from planning, and given that articles from those other (larger) disciplines often tend to get cited more than articles from small disciplines like planning, it is possible that high citation counts are correlated with large numbers of co-authors at the level of disciplines, but perhaps less so within a given discipline.
We also found that making an electronic copy of the paper freely accessible online does not appear to increase citation counts, even though freely accessible papers are (by definition) easier to acquire and read than papers that are not freely accessible. One potential explanation for this unexpected finding is that planning researchers (many of whom are likely to be affiliated with a university) might already have free access to electronic copies of journal articles through their university library, such that making a paper freely accessible online (e.g., through ResearchGate) might not actually make papers any more accessible than they already are for many potential citers.
There is a long-standing debate in the social sciences about the relative merits of quantitative versus qualitative research methods (Johnson and Onwuegbuzie 2004). In the planning field, Du Toit, Boshoff, and Mariette (2017) found that papers that used quantitative methods receive more citations on average than papers that use qualitative or mixed methods, but they did not control for potential confounding factors and their findings were based on papers that were published in a single planning journal (i.e., Journal of Planning Education and Research). Our findings help to improve our understanding of the relative value that planning scholars place on quantitative and qualitative methods by showing that there is no significant difference in citation counts for articles that use quantitative versus qualitative or mixed methods, once a broader set of planning-related journals have been taken into account and several other factors have been controlled for.
Whereas past studies have found that different features of article titles help to determine article citation counts, we found that neither title length, presence of punctuation marks, phrasing in the form of a question, nor reporting the research design in the title appears to influence citation counts in planning. Other paper-related factors also do not seem to matter, including whether or not the paper contains an appendix and the number of figures and tables that the paper contains.
We end our discussion of findings with a caution that our findings might or might not be generalizable to all planning scholars. Given that our sample of authors possibly overrepresents more senior researchers and underrepresents more junior researchers, it is possible that our findings apply more to senior researchers than to junior. Future research can help to resolve this uncertainty by exploring whether the factors that determine citation counts depend at all on the seniority and past productivity of article authors.
Concluding Thoughts on Citation Counts and Meta-Research in Planning Scholarship
Recent meta-research aimed at better understanding planning scholarship has focused on citation counts. We have expanded on these recent studies by showing that citation counts for articles in planning-related journals appear to be influenced by certain journal, author, and paper-related factors. Our findings show that the planning literature behaves in some ways that are similar to and in some ways that are different from other literatures, in the sense that some factors that appear to influence citations in other fields also appear to influence citations in planning while other factors do not. Our most noteworthy finding is arguably that the diversity of specializations represented in planning has important implications for planning scholars and users of citation counts, in that papers about some topics tend to attract more citations than others.
Planning researchers can use our findings to help them make strategic choices aimed at increasing the visibility and potential impact of their research articles. When researchers have a manuscript that is ready for submission to a journal, for example, they might wish to select a journal based on its impact factor rather than its reputation among planning scholars, other things being equal. Planning researchers also might benefit from being thorough in addressing their topic and in citing other works in their papers, whether they are writing literature review papers or not. However, we do not advise that planning researchers select to write about particular topics simply because those topics might attract more citations. Less cited topics like diversity in planning and planning education are important enough to warrant study even if they might not lead to highly cited papers, and it is worth noting that papers can still be influential even when they are not highly cited.
We noted earlier that citation counts are commonly used to evaluate the productivity and impact of university scholars and programs, based on a belief that higher citation counts indicate higher productivity/impact. Our findings suggest, though, that while citation counts might be easy to measure, they are not so easy to interpret. When citation counts are used in planning to make evaluations about particular scholars, for example, evaluators might get more meaningful results if they were to make adjustments for the topics that the scholars focus on in their research. In particular, citation counts for scholars who study less popular (but nevertheless important) topics could be adjusted upward, or the counts for scholars who study more popular topics could be adjusted downward. The important point is to recognize that two scholars with different total citation counts might actually be equally productive and influential, if they happen to study different topics in planning. Until and unless the planning community makes a decision that some topics are more worthy of study than others, then researchers who choose to study less popular topics should not be overly penalized if their articles have relatively low citation counts.
Last, we hope that our study will inspire more meta-research on planning scholarship, with a goal of enhancing what we do as planning researchers and the extent to which we can influence planning theory and practice. Planning is a complex discipline that uses a wide range of methods to study a wide range of topics. It is easy to get lost in the trees, while losing sight of the forest. We encourage other planning researchers to add their own contributions to the small but growing body of research that seeks to better understand our complex discipline and to make sure that it remains visible and relevant for practitioners, scholars, and other users of planning knowledge.
Supplemental Material
Appendix_online_supp – Supplemental material for Why Do Some Articles in Planning Journals Get Cited More than Others?
Supplemental material, Appendix_online_supp for Why Do Some Articles in Planning Journals Get Cited More than Others? by Mark R. Stevens, Keunhyun Park, Guang Tian, Keuntae Kim and Reid Ewing in Journal of Planning Education and Research
Supplemental Material
Correlation_Matrix_online_supp – Supplemental material for Why Do Some Articles in Planning Journals Get Cited More than Others?
Supplemental material, Correlation_Matrix_online_supp for Why Do Some Articles in Planning Journals Get Cited More than Others? by Mark R. Stevens, Keunhyun Park, Guang Tian, Keuntae Kim and Reid Ewing in Journal of Planning Education and Research
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Notes
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
