Psychology and Wikipedia: Measuring Psychology Journals’ Impact by Wikipedia Citations

Abstract

The aim of the study was to explore the impact of peer-reviewed psychology journals on Wikipedia articles. We are presenting a rank of academic journals classified as pertaining to psychology, most cited on Wikipedia, as well as a rank of general-themed academic journals that were most frequently referenced in Wikipedia entries related to psychology. We then compare the list to journals that are considered most prestigious according to the SciMago journal rank score. Additionally, we describe the time trajectories of the knowledge transfer from the moment of the publication of an article to its citation in Wikipedia. We propose that the citation rate on Wikipedia, next to the traditional citation index, may be a good indicator of the work’s impact in the field of psychology.

Keywords

psychology journals citations Wikipedia psychology in online media publication impact

Journal quality and rankings have been a topic of recurring debates in psychology research (Mayor, 2010; Paulus et al., 2018). Some researchers approached this issue as early as in the 1970s by studying the ranking of journals by American Psychology Association (APA) members (Koulack & Keselman, 1975). There have been a number of studies using different measures of impact, including the impact factor, over the years as well (Feingold, 1989; Rushton & Roediger, 1978; White & White, 1977), pinpointing various problems of journal rankings in psychology (Boor, 1982; Christensen-Szalanski & Beach, 1984; Over, 1978) and revisiting the issue of how to use them (Fehr et al., 1981; Stamps & Fehr, 1980).

The publications have relied on some form of professional evaluation, either directly, through professional associations’ members polling or, indirectly, through citations in other academic publications. In this article, we take a different approach: We analyze which psychology journals are most commonly referred to in articles in Wikipedia, the most popular and largest online encyclopedia developed by volunteers. We also study which journals are referred to in articles about psychology in Wikipedia. By doing this, we return to the conversations that have been important for the academic community of psychology researchers but that have not moved forward significantly over the past decades.

We present an analysis of academic journals used in Wikipedia articles about psychology, comparing the results to the existing established measures, such as the SciMago journal rank (SJR), and interpret the results and conclusions for psychology as a field. Our study is a large data set computational bibliometric analysis, and as such, it does not build on a social theory. However, it brings important results to our knowledge of the social impact of computing on academic work, directly within the scope of Social Science Computer Review.

Our results are significant for at least two reasons. First, Wikipedia is the most read source of academic knowledge worldwide. Journals commonly used there are by default quite likely more successful in reaching the general audience than the others, as Wikipedia is the first-stop for scientific information (Teplitskiy et al., 2017). Thus, learning which journals result in the largest general public impact is of high interest for academic psychologists. Second, since Wikipedia relies on contributions from nonexperts, and is curated by a large worldwide nonexpert community, which somewhat is a proxy for the general public, it is extremely interesting to discover what contributions they consider relevant in terms of sources of knowledge about psychology. This result should be of wide interest to psychologists as a profession.

Since discovering ways of reaching the general public with psychology science has been a topic of intensive research (Kaslow, 2015), just as much as the appeal of certain topics for the general public (Beck, 2010), our findings are a contribution to these conversations. Additionally, we already know that general science impact should be distinguished from impact in psychology (Hegarty & Walton, 2012). Measuring which psychology journals are most commonly reflected in a general audience encyclopedia, as well as which journals are most commonly used in articles on psychology topics, develops this important topic further.

Journal Rankings

Journal rankings are a topic of hot debate in many fields. For some scholars, they are a great example of Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure, p. 308” (Strathern, 1997) and a way of perpetuating inequalities (Davidson, 2019). Imbued with conundrums undermining the purpose of social scientific research (Özbilgin, 2009), they are still appealing to the academic milieu (Nkomo, 2009): After all, they allow using a simple heuristics of evaluating published research just by checking against a ranking table instead of reading it.

According to some scholars, journal rankings are a powerful tool of exerting bureaucratic control over faculty through strict measures (Bok, 2009; Jemielniak & Greenwood, 2015; Martin-Sardesai et al., 2017), admittedly as well, journal rankings entail ranking games and shift focus from doing research to publishing (Osterloh & Frey, 2015). In doing so, they may result in political struggles and detour the professoriate from their mission to reliably communicate the research outcomes to the general public (Macdonald & Kam, 2007) and derail research public funding by incentivizing publishing research results in outlets that may be linguistically and economically not available to the domestic readers (Neff, 2018). According to some findings, journal rankings may also contribute to suppressing interdisciplinary research (Rafols et al., 2012) by prioritizing journals that have a narrow scope as well as suppress diversity of ideas in psychological science (De Rosa, 2017).

Some critics also raise that the most popular journal rankings may be prone to a significant range of errors, technical and methodological flaws, or various biases (Amin & Mabe, 2000; Ferrer-Sapena et al., 2016; Vanclay, 2012), as well as interdisciplinary variations (Leydesdorff & Opthof, 2010). The lack of transparency in the fine details of the methodology applied each year contributes to the perception of a lack of academic rigor of such measures and encourages manipulation (Archambault & Larivière, 2009; Moustafa, 2015). Also, relying on citations alone as a measure of quality is prone to the error of not distinguishing between building on some work and criticizing it. There are also some indicators that the journal impact factor (JIF) alone is a poor reflection of scientific quality (Brembs et al., 2013). Moreover, gaming the system is quite widespread. For instance, coercing journal self-citation is common (Humphrey et al., 2019) and so are citation cartels (Fister et al., 2016; Perez et al., 2019). These problems spur some journals to remove JIF information from their websites (Casadevall et al., 2016), the use of JIF as a measure for evaluating research has been criticized at least over the last 20 years (Seglen, 1997), and focusing overly on citation metrics is openly called an aberration (Moustafa, 2016). Yet, factor-mania may be a variation of the “tragedy of the commons”: Even though the consequences are detrimental to science as a whole, individual scholars act rationally in their self-interest perpetuating it (Casadevall & Fang, 2014).

In spite of the wide and often justified criticism, journal rankings also have some undisputable practical merits, even beyond serving as a rough estimate of academic quality. For instance, they may be useful to the members of the academic community by offering them a method of distinguishing between predatory and nonpredatory journals (Demir, 2018; Sorokowski et al., 2017). Even more importantly, they are a simple, externally verifiable measure of the average impact an article published in the journal can make. Given the fact that science advances through dialogue with other researchers, which is reflected through citations and building upon someone else’s work, citation rankings are one of the very few available measures of possible influence on the body of academic knowledge as a whole.

Irrespective of one’s opinion on the usefulness of the rankings, since they become proxies for assessing the quality of individual work (Eacott, 2018), there is an increasing pressure on journals to fare well in them. Their importance is also growing (Tadajewski, 2016). For instance, bibliometric indicators are proliferating in tenure reviews, hire-or-fire decisions and have spread in most academic fields (Kulczycki, 2019; Rushforth & de Rijcke, 2015). Psychology journals also react to the dynamic increase in access to information and the popularity of journal rankings (Adair & Vohra, 2003). One of the effects is that especially low- and middle-impact journals are often inflated with self-citations, while the top ones are less affected by it (Anseel et al., 2004).

One of the possible reasons for this phenomenon may be that the alternatives to mainstream journal rankings are not widely available. Even though some researchers discuss the possibility to rely on consensus-based evaluations (Aledo et al., 2018), and there are also some metrics that include, for example, social networks’ popularity of publications (Papakostidis & Giannoudis, 2018), they are much less universally recognized or accepted. Some scholars also consider refining methodologies of journal evaluations (Pagani et al., 2015). It is quite clear that increasing the number of available, sensible methodologies of ranking journals may at least partly restore the balance in these metrics (Okumus et al., 2018). One possible alternative that we would like to discuss in this article could be checking which psychology journals are most commonly referred to in Wikipedia. Additionally, it is quite clear that any single journal ranking is prone to certain kinds of biases and errors inherent to its methodology. However, combining different measures may get us closer to a more diverse and balanced reflection of academic publication impact. For that purpose, we believe that supplementing measures such as IF or SJR with the measurement of popular use in a peer-produced encyclopedia is highly valuable.

Wikipedia: A Crowd-Sourced Encyclopedia

Wikipedia was established in 2001 and has quickly become one of the most popular websites in the world, currently ranked in the top 10 in TopSites ranking by Alexa.com. Its success can be partially attributed to the fact that it is “an encyclopedia that anyone can edit.” It relies on the wisdom of crowds (Surowiecki, 2004) through a largely self-governed, a-hierarchical, and spontaneous cooperative organization (Jemielniak, 2016; Konieczny, 2017). As a large-scale collaboration project, it is successfully crowd-sourcing knowledge dissemination (Reagle, 2010; Uhlmann et al., 2019).

Although Wikipedia communities in different languages have developed elaborate systems of rules and policies over the years (Jemielniak, 2014), including detailed conduct and civility norms, as well as very specific regulations about notability, the rules about source inclusion are quite straightforward: They have to be verifiable and reliable (Niederer & van Dijck, 2010). This is so to reduce the need for trusting the editors and to transfer this trust onto procedures so as to fully capitalize on the collective intelligence design of peer production (Benkler et al., 2015). These rules do not prevent, for example, hindsight bias (Oeberst et al., 2018), but they enforce using proper academic sources for scientific topics.

Even though the standards of quality vary between different Wikipedia projects (Jemielniak & Wilamowski, 2017), the overall quality of Wikipedia articles is surprisingly high (Michelucci & Dickinson, 2016), even in highly specialized topics (London et al., 2019). As early as in 2005, a study published in Nature described Wikipedia as going “head to head” with Britannica (Giles, 2005). Since then Wikipedia has grown 6-fold in terms of the number of articles and is currently over 85 times the size of 120-volume Encyclopedia Britannica by word count (Wikipedia, 2019). The overall accuracy of Wikipedia is on par with the professional sources (Chesney, 2006; James, 2016), and a systematic overview of academic research on the topic confirms that Wikipedia’s quality and accuracy are high (Mesgari et al., 2015) even though errors of omission for older topics are frequent (Brown, 2011).

In psychology, the topical coverage on Wikipedia was already relatively comprehensive a decade ago (Schweitzer, 2008). In some studies, Wikipedia proved to be as reliable as or more reliable than the Britannica or psychiatry textbooks although it was rated lower for readability (Reavley et al., 2012).

Clearly, even though Academia still treats Wikipedia with suspicion (Aibar et al., 2015; Jemielniak & Aibar, 2016; Konieczny, 2016), its accuracy is already sufficient for most purposes, including higher education, and Wikipedia editing is becoming increasingly more popular as a course assignment in psychology classes (Shane-Simpson et al., 2016). This is possibly due to the call from APA to actively participate in Wikipedia editing (Breckler, 2010) or a similar initiative by the Association for Psychological Science (Banaji, 2011).

These facts make it even more interesting to discover which psychology journals are used on Wikipedia most frequently, and which references are cited in Wikipedia articles on psychology. While the usage of academic journals Wikipedia-wide may be a measure of the journal’s overall popularity and, quite likely, is biased toward general science publications, as well as toward journals with a popularizing mission, the references in articles specifically covering psychological concepts may be a good reflection of what publications are perceived as best addressing them.

Research Questions

The aim of our study was to check which psychology journals are most often cited in Wikipedia (1). Also, we wanted to know which academic journals are frequently referenced in Wikipedia articles on psychology topics (2). Additionally, we were interested in understanding the similarities and differences of such ranking of popularity, when compared to a standard journal ranking measures SJR. We also wanted to observe patterns related to the frequency of citations, the trajectory of citations in time, distribution of publication dates, and the time between the publication date of an article and the reference created in Wikipedia.

Method

We conducted a two-stage study. First, we checked which psychology journals are referred to in the entire English Wikipedia. To do that, we processed the full repository of Wikipedia articles with a self-developed script and cross-referenced it with a SciMago database of academic journals in the field of psychology, that is, 1,360 titles of academic journals classified as psychology journals. It should be noted that we relied on SciMago classification rigorously: In other words, we avoided any judgment calls. As a result, for example, some psychiatry journals have been included, as they are classified also as psychology in SciMago, and some have not. We consider this to be a feature rather than a flaw in our design. We looked for DOI and ISSN matches and also processed journal names if these identifiers were missing in Wikipedia. This way, we obtained a list of journal titles with their number of citations in Wikipedia.

Second, we checked which academic journals are referred to in Wikipedia in articles specifically related to psychology. For that purpose, we used 1,763 entries listed in the “index of psychology articles,” curated by WikiProject Psychology, a group of volunteer Wikipedians working in the area of psychology. The list is a set of psychology topics, prechosen by the Wikipedia community as being the core of knowledge about psychology. We then scraped the data containing the number of references to the journals listed in these articles.

For a comparison to existing journal citation metrics, we decided to use SJR (developed by Elsevier) for several reasons. First, even though both SJR and JIF (impact factor based on Journal Citation Reports [JCR], curated by Clarivate) rely on the number of citations to a given journal, SciMago is a database that contains roughly 3 times more titles than JCR, and we wanted to cover a larger data set of academic journals. Second, SJR is a free database, while JCR requires a subscription. Additionally, it is normalized within disciplines and thus allows cross-disciplinary comparisons, while JCR is not. Finally, there is research suggesting that SJR may be a better overall proxy of academic impact (Bollen et al., 2009). It was developed specifically to address some criticism that JIF received (Butler, 2008) and incorporates a measure of “inheriting prestige,” that is, citations in higher ranked journals are worth more than in the ones ranked lower, similarly to Google PageRank.

Results

In the entire body of English Wikipedia, we found 35,609 references to 357 psychology journals. The list of the 30 most cited psychology journals in the whole collection of English Wikipedia articles is presented in Table 1. The most frequently cited psychology journals are the Journal of Personality and Social Psychology, American Psychologist, and Psychological Bulletin. The three most cited journals make up 11% of all the references to psychology journals in Wikipedia, whereas the 10 most cited journals constitute as much as 24% of all the references. Quite clearly, journals that operate on a meta-level, that address the topics in the psychology field interesting predominantly to academic psychologists, are less popular.

Table 1.

Psychological Journals Most Cited in Wikipedia.

Journal Title	Number of Citations	Absolute Rank	Rank When Weighted
Journal of Personality and Social Psychology	1,976	1	3
American Psychologist	1,064	2	7
Psychological Bulletin	1,038	3	2
Psychological Science	872	4	10
Psychological Review	853	5	1
Archives of Sexual Behavior	668	6	30
Personality and Individual Differences	650	7	137
Child Development	573	8	26
Trends in Cognitive Sciences	504	9	15
Neuroscience & Biobehavioral Reviews	485	10	69
Journal of Applied Psychology	478	11	11
Developmental Psychology	472	12	37
Neuropsychologia	471	13	96
Psychological Medicine	398	14	84
Personality and Social Psychology Bulletin	398	15	25
Sex Roles	396	16	21
Behavioral and Brain Sciences	378	17	97
Cognition	369	18	70
Physiology & Behavior	369	19	152
Frontiers in Psychology	368	20	354
Current Directions in Psychological Science	352	21	13
Journal of Affective Disorders	350	22	240
Annual Review of Psychology	341	23	5
Journal of Experimental Social Psychology	335	24	35
Journal of Consulting and Clinical Psychology	332	25	27
Intelligence	331	26	18
Clinical Psychology Review	322	27	16
Journal of the American Academy of Child & Adolescent Psychiatry	311	28	63
Journal of Child Psychology and Psychiatry	290	29	53
Journal of Autism and Developmental Disorders	277	30	173

Since psychology journals might potentially be used as a source of knowledge about topics that are not necessarily strictly psychology-related, we then explored references to any journals in Wikipedia articles that had been classified as psychology-themed. The articles included 7,848 references to 360 journals. The rank of the 30 most often cited journals is presented in Table 2. The top three journals referenced in psychology articles in Wikipedia are the Journal of Personality and Social Psychology, SSRN Electronic Journal, and Psychological Bulletin. The three most frequently referenced journals within the psychology-themed articles constitute 13% of all the references within this article collection, whereas the top 10 cover 28% of all the references. In both of the rankings, the most cited journal remains the Journal of Personality and Social Psychology, with 1977 references on all articles on Wikipedia, and 463 when only articles categorized as psychology-themed are taken into account. We see that many well-established psychology journals may be referred to in encyclopedic entries on different aspects of human behavior, not necessarily strictly pertaining to psychology.

An absolute number of citations is a measure of a journal’s prominence, but it does not account for an average impact of one article published in it—after all, mega-journals, publishing thousands of articles per year, could be reflected more often just because of the sheer number of publications, even if their average quality was lower than in the journals publishing just a dozen articles each year. In order to address that issue, we calculated the ranking of journals weighted by the number of articles published per year (including only journals with at least 10 publications over the last three years, to eliminate artifacts). The purpose of this approach was to see whether the journals publishing thousands of articles per year do not dominate the ranking because of the sheer quantity rather than the pure quality of publications.

The most frequently cited psychology journals, when they are weighted by the annual number of articles published, are Psychological Review, Psychological Bulletin, and Journal of Personality and Social Psychology. The differences in our ranking after adjusting for the number of publications per year are largely dependent on the way weights are applied. In our approach, with a linear number of publications used, they definitely provide an additional, nonreducible point of view. We believe that this analysis adds an important perspective on evaluating the journal’s impact.

We prepared a similar weighted analysis for journals referenced in psychology-themed Wikipedia articles. The top three journals according to this criterion are Psychological Bulletin, Psychological Review, and Annual Review of Psychology. This additional analysis eliminated SSRN Electronic Journal from the top of the ranking, which we consider a valuable re-adjustment. Interestingly, taking the number of articles published per annum also significantly lowered the rank for journals such as Science, Cochrane Database of Systematic Reviews, PNAS, Nature, PLos ONE, or BMJ, which publish large numbers of articles. We interpret this result as indicating that the weighting process proves useful and adds an important perspective.

Table 2.

Journals Most Cited in Wikipedia Articles Classified as Psychology-Themed.

Journal Title	Number of Citations	Absolute Rank	Rank When Weighted
Journal of Personality and Social Psychology	463	1	4
American Journal of Psychiatry	342	2	9
SSRN Electronic Journal ^a	327	3	—
Psychological Bulletin	265	4	1
Science	246	5	522
Cochrane Database of Systematic Reviews	242	6	292
Psychological Review	208	7	2
JAMA Psychiatry	198	8,5	43
American Psychologist	198	8,5	10
Nature	192	10	895
Proceedings of the National Academy of Sciences	177	11	1,117
Psychological Science	176	12	17
The Lancet	173	13	569
British Journal of Psychiatry	158	14	45
BMJ	136	15	1,127
Personality and Individual Differences	130	16	272
Child Development	128	17	35
Clinical Psychology Review	119	18	8
The Journal of Clinical Psychiatry	113	19	91
Archives of Sexual Behavior	112	20,5	85
Psychological Medicine	112	20,5	119
Personality and Social Psychology Bulletin	107	23	24
PLoS ONE	107	23	2,598
Annual Review of Psychology	107	23	3
Biological Psychiatry	105	25	141
Developmental Psychology	99	26	70
Journal of Affective Disorders	94	27	481
Journal of Consulting and Clinical Psychology	91	28	25
Trends in Cognitive Sciences	90	29	33
Journal of Abnormal Psychology	89	30	19

^a SSRN Electronic Journal is a nonreviewed paper repository.

Coverage

It is striking that from among 1,350 psychology journals, only 357 are cited at least once in the entire Wikipedia. This shows that editors creating content in the world’s most popular encyclopedia refer to only about 26% of content-specific, academic resources available. Additionally, it is very surprising that open access journals do not make it to the top 30, with the notable exception of Frontiers in Psychology in the group of psychology journals most cited on Wikipedia as a whole, and SSRN Electronic Journal in the group of academic journals most referred to in Wikipedia entries classified as psychology-themed. In a study of medical journals on Wikipedia (Jemielniak et al., 2019), the open-access effect was more noticeable and explained by Wikipedia editors’ preference for open knowledge. Clearly, this preference is not predominant for editors who add information on psychology to Wikipedia—possibly due to the articles being more widely available on academic sharing websites, such as Researchgate.net or Academia.edu. The high position of SSRN Electronic Journal is interesting, and it may denote that Wikipedia editors do not recognize that it is a preprint, nonreviewed paper repository. However, as our additional analysis of citations after weighting the annual number of articles published by a journal shows, SSRN Electronic Journal high position in absolute numbers ranking is largely resulting from its massive number of non-peer-reviewed papers stored in the repository each year.

Nature, Science, PNAS, and PLos ONE are general focus mega-journals, and it is not surprising that they are top-cited in psychology-themed articles in absolute numbers. Their position is largely a result of the number of publications per year, as we can observe in the weighted ranking.

Correlation Between SJR and Wikipedia Citation Rate

In order to check whether there is a correlation between the number of citations the psychology journals get in Wikipedia and the journals’ SJR, we ran a correlation test in SPSS version 25. Results of the Pearson correlation indicated that there is a significant positive association between SJR measured by 2017 SciMago Ranking and Wikipedia citation rate, r = .436, p < .001, n = 285.

Interdisciplinary Character

Since it might be expected that journals that tackle a variety of broad topics may generate more citations in a source aimed at the general public than journals with narrow scope related to very specific subjects, we looked at how interdisciplinary character of the journal is aligned with the Wikipedia citation rank—or, in other words, to see whether being highly cited in the field of psychology is a good proxy of being highly cited in articles on Wikipedia from other disciplines. This is presented in Figure 1, where the value rank of the journals most frequently cited in Wikipedia are plotted, with the x-axis representing the rank based on the number of citations (the lowest rank represents the highest number of citations) of the journal within Wikipedia psychology-themed articles and the y-axis represents the rank values of all academic journals based on their citations in Wikipedia.

Figure 1.

Correlation between SJR and Wikipedia citation rate.

As may be observed in Figure 2, the references in the two categories have a similar trajectory. This means that journals that are well-cited across psychology-related Wikipedia entries are also generally well-cited across the whole Wikipedia.

Figure 2.

Wikipedia citation rank.

Based on the Wikipedia citation rank values, we have calculated the rank difference by subtracting the value of Wikipedia citation rank of journals which were referred to in psychology articles on Wikipedia from the value of the general Wikipedia citation rank, that is, the rank based on all citations of a journal in Wikipedia. If the difference is negative, the larger the negative values, the larger the relative interdisciplinary character of the journal. If the difference is positive, the larger the difference value, the more psychology-themed the journal. The Wikipedia citation rank differences of the top 70 academic journals are presented in Figure 3.

Figure 3.

Journals’ Wikipedia citation rank differences.

Change in Citations in Time

We then traced the number of citations of the top 15 journals in Wikipedia in time. The number of citations in Wikipedia psychology-themed entries is shown in Figure 4, whereas the number of citations to psychology journals in all Wikipedia entries is shown in Figure 5.

Figure 4.

The number of citations to journals in Wikipedia psychology-themed articles.

Figure 5.

The number of citations to psychology journals in all Wikipedia articles.

Interestingly, the trajectories of the journals both for all-Wikipedia and for psychology-articles-only are relatively similar. We observe increasing value with time, and the incline seems steeper for a number of journals in 2011–2012.

The trajectory of the Journal of Personality and Social Psychology is steeper than the one of any other journal. The arising question is whether this might be explained by any changes within the journal productivity such as the onset of a higher number of articles published shortly before the trend started. This does not seem to be the case: the journal has been steadily producing six issues a year, with a similar number of articles each year. Another explanation might be the circularity of the citation process. Journals that are often referenced get more citations since the references are often the sources of information (Ball, 2002). This could potentially clarify the effect of cumulative advantage observed here for Journal of Personality and Social Psychology. We believe that the spike in Journal of Personality and Social Psychology may to some extent be related to the replication crisis in psychology (Stroebe & Strack, 2014), and the fact that a number of articles, which have had their replicability questioned, have been published in that journal. Speaking more broadly, given the fact that, arguably, the replication crisis has affected social psychology the most, it is not surprising that Journal of Personality and Social Psychology, the flagship journal of the field, has received more attention over the last years.

We ran additional analyses for journals weighted by the number of articles published per annum. The time analysis for the number of citations to journals in Wikipedia psychology-themed articles weighed by the number of articles published per annum is presented in Figure 6, and the number of citations to psychology journals weighted by the number of articles published per annum in all Wikipedia articles is shown in Figure 7. The Journal of Personality and Social Psychology trajectory is less prominent when we adjust for the number of articles per annum although it is still quite steep. In both of our weighted rankings, two journals are strong outliers: Psychological Review and Psychological Bulletin. This result indicates that the articles published in them are considered to be of top quality and relevance universally in psychology-themed articles and among psychology journals cited in all Wikipedia articles.

Figure 6.

The number of citations to journals in Wikipedia psychology-themed articles weighed by the number of articles published per annum.

Figure 7.

The number of citations to psychology journals weighted by the number of articles published per annum in all Wikipedia articles.

Trajectories of Transfer From Academic Journals to Wikipedia

Our next step was to check how much time passes between the publication of an article and its citation on Wikipedia. This is illustrated in frequency charts in Figure 6.

There is variability across journals in the time needed for an academic article to be cited in Wikipedia (see Figure 8). Whereas texts from Archives from Sexual Behavior, Psychological Science, or Personality and Individual Differences make their way into Wikipedia relatively quickly, with the number of citations decreasing with time, articles from other journals might either increase in citations over time (Journal of Personality and Social Psychology), have a steady citation rate (American Psychologist), or show a somewhat irregular pattern. Across the sample of the nine most cited journals, it takes on average 6,176 days for an article to be referenced in a Wiki article (SD = 5,925), including works published before Wikipedia started. We explain the speed of the propagation of research from Sexual Behavior, Psychological Science, or Personality and Individual Differences by the topic coverage relatively more interesting for lay readers. This result indicates that article readership and coverage is depending on the popularity of the topic covered, which is not surprising, but at the same time serves as an important reminder that scientific excellence should not be equaled with citations only.

Figure 8.

The time between the publication of articles in academic journals and their citation on Wikipedia.

The distribution of the number of journal references within one Wikipedia article, depicted in Figure 9, is highly asymmetric (M = 16.9, SD = 23, min = 1, max = 300).

Figure 9.

The number of articles (y-axis) with the number of academic articles referred (x-axis). Note. For readability, we truncated the figure. Articles containing more than 100 journal references were cut (1.6% of all analyzed Wikipedia articles).

Limitations

One important limitation our study is prone to is a lack of distinguishing between affirming and contesting character of references. A publication that is highly criticized receives a lot of citations even though they are not indicative of scientific quality. It is a flaw that is currently inherent to all citation metrics, including JCR (impact factor) and SJR, and one that cannot be easily resolved in a large data set computational study. We argue that while the problem is definitely real on the level of a specific article (or a person), it does not persist on the level of a journal. In other words, while it is possible that a single person writes an academic article that receives a lot of citations as a result of criticism, it is highly unlikely that there is a journal which consistently receives a lot of attention and criticism from other reputable journals because of publishing questionable results. However, this limitation should be kept in mind.

Our study relies on journals in the field of psychology. While they are valid within this discipline, we do not know whether similar robust results are observable in other fields.

Discussion

Our aim was to check which psychology journals are most often cited in Wikipedia and which academic journals are most frequently referenced in psychology articles of Wikipedia. We were surprised to find that only 357 of 1,350 are cited at least once in the entire Wikipedia. An interesting finding was that an effect of accumulative advantage may be observed in Wikipedia citation rate, that is, there is a pattern that journals that were cited a lot, get even more citations in time, whereas journals that only have a few citations grow at a very limited pace. This might be explained by the potential practice of people referring to sources that already have their citations in Wikipedia and using a reference list in a Wikipedia article as a guide for their further study, which later results in another Wikipedia article. Another possible explanation, requiring further research, is that there may be a core group of Wikipedia editors developing articles on psychology, who have strong preferences or default to the journals they already know.

Importantly, we have found a significant correlation between the Wikipedia citation rate of psychology journals and the journals’ SJR score. This is not surprising. However, while both SJR score and accessibility (open access) have been shown to be strong predictors of an article being referenced on Wikipedia (Teplitskiy et al., 2017), the latter (open access) has not proven true for psychology journals. It is also worth noting that the journals most cited in Wikipedia are APA or APA-affiliated journals (all top five journals according to our ranks, apart from SSRN Electronic Journal). This may be related to the general trust of the community to the organization which is known for producing and disseminating knowledge about psychology. We interpret the citability also as a function of the popularity of a given topic. For instance, online news about social psychology is generally more popular than other psychology topics (McPhetres, 2019). This may partly additionally explain the high position of the Journal of Personality and Social Psychology in the ranks.

We have found that journals that are well-cited across psychology-related Wikipedia entries are also generally well-cited across the whole of Wikipedia. This is an interesting finding, arguing against the expectation that journals with a narrowed scope would be cited more often in psychology articles, but not across the whole Wikipedia, whereas journals whose scope is broader, hence the topics covered there are of more interdisciplinary nature, would not be cited as often in psychology journals as they would across the whole Wikipedia. It is possible that the circularity of referencing may play a role in this result.

We were also interested in the pattern of citations of particular journals, as well as the trajectories that information has from the publication date of an academic journal to the citation of the article in Wikipedia. We found that the patterns vary across the most cited journals. It may be expected that the access policies of a particular journal may contribute to the creation of the patterns to some extent, but other interpretations are possible. One interpretation is that, again, the specific content may generate differential public interest, which in turn influences the speed of transferring the research results to texts popularizing them. This hypothesis aligns with the trajectories of Archives of Sexual Behavior—a topic that is, possibly, of broad public interest and that is tracked and described soon after the publication of the original research report.

Studying the aggregated data to see which journals are most effective at reaching the general public may be used as an interesting supplementary way of evaluating journals’ actual impact on knowledge dissemination in society. Online mentions of academic works show positive medium and high correlations to more traditional metrics such as impact factor when measured on the level of psychology as a whole field (Vogl et al., 2018). We propose that the citation rate on Wikipedia, next to the traditional citation index, may be a good indicator of the work’s impact.

We propose that future research directions may include studying different disciplines’ citations on Wikipedia, so as to determine the interdisciplinary variance. Additionally, we believe it would be useful to distinguish positive and negative citations when it is technologically viable, to increase the granularity of the results. One more possible path for future studies is discovering the average time between the publication date and the citation varying across disciplines. Finally, while difficult, we also believe it would be worthy to look for measures of a given article being accessible through university repositories or academic social networking websites such as Academia.edu or Researchgate.net, as it affects its possible reach and citations on Wikipedia.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a research Grant from a research Grant Number PPN/BEK/2019/1/00233/U/00001 from the Polish National Agency for Academic Exchange.

References

Adair

J. G.

Vohra

(2003). The explosion of knowledge, references, and citations. Psychology’s unique response to a crisis. The American Psychologist, 58 (1), 15–23.

Aibar

Lladós-Masllorens

Meseguer-Artola

Minguillón

Lerga

(2015). Wikipedia at university: What faculty think and do about it. The Electronic Library, 33 (4), 668–683.

Aledo

J. A.

Gámez

J. A.

Molina

Rosete

(2018). Consensus-based journal rankings: A complementary tool for bibliometric evaluation. Journal of the Association for Information Science and Technology, 69(7), 936–948.

Amin

Mabe

(2000). Impact factors: Use and abuse. International Journal of Environmental Science and Technology, 1(1), 1–6.

Anseel

Duyck

De Baene

Brysbaert

(2004). Journal impact factors and self-citations: Implications for psychology journals. The American Psychologist, 59(1), 49–51.

Archambault

É.

Larivière

(2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649.

Ball

(2002). Paper trail reveals references go unread by citing authors. Nature, 420 (6916), 594.

Banaji

(2011). Harnessing the power of Wikipedia for scientific psychology: A call to action. Association for Psychological Science. https://www.psychologicalscience.org/observer/harnessing-the-power-of-wikipedia-for-scientific-psychology-a-call-to-action

Beck

D. M.

(2010). The appeal of the brain in the popular press. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 5(6), 762–766.

10.

Benkler

Shaw

Hill

B. M.

(2015). Peer production: A form of collective intelligence. In Malone

(Ed.), Handbook of Collective Intelligence. MIT Press.

11.

Bok

(2009). Universities in the marketplace: The commercialization of higher education. Princeton University Press.

12.

Bollen

Van de Sompel

Hagberg

Chute

(2009). A principal component analysis of 39 scientific impact measures. PLoS One, 4(6), e6022.

13.

Boor

(1982). The citation impact factor: Another dubious index of journal quality. The American Psychologist, 37(8), 975.

14.

Breckler

(2010). Don’t like Wikipedia? Change it. American Psychological Association. https://www.apa.org/science/about/psa/2010/12/wikipedia-change

15.

Brembs

Button

Munafò

(2013). Deep impact: Unintended consequences of journal rank. In Frontiers in Human Neuroscience (Vol. 7). https://doi.org/10.3389/fnhum.2013.00291

16.

Brown

A. R.

(2011). Wikipedia as a data source for political scientists: Accuracy and completeness of coverage. Political Science & Politics, 44(02), 339–343.

17.

Butler

(2008). Free journal-ranking tool enters citation market. Nature, 451(7174), 6.

18.

Casadevall

Bertuzzi

Buchmeier

M. J.

Davis

R. J.

Drake

Fang

F. C.

Gilbert

Goldman

B. M.

Imperiale

M. J.

Matsumura

McAdam

A. J.

Pasetti

M. F.

Sandri-Goldin

R. M.

Silhavy

Rice

Young

J.-A. H.

Shenk

(2016). ASM journals eliminate impact factor information from journal websites. mSystems, 1 (4), e00088–16.

19.

Casadevall

Fang

F. C.

(2014). Causes for the persistence of impact factor mania. mBio, 5(2), e00064–14.

20.

Chesney

(2006). An empirical examination of Wikipedia’s credibility. First Monday, 11(11), >http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/1413/1331 (retrieved on 19 June 2012)

21.

Christensen-Szalanski

J. J. J.

Beach

L. R.

(1984). The citation bias: Fad and fashion in the judgment and decision literature. The American Psychologist, 39(1), 75–78.

22.

Davidson

E. J.

(2019). A reflection on “Information systems research: Thinking outside the basket and beyond the journal.” Communications of the Association for Information Systems, 45(1), 11.

23.

Demir

S. B.

(2018). Predatory journals: Who publishes in them and why? Journal of Informetrics, 12(4), 1296–1311.

24.

De Rosa

(2017). Will analytics suppress a diversity of ideas in psychological science? Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 12(6), 1138–1140.

25.

Eacott

(2018). Fluctuations in value: The shifting worth of journal articles in educational administration. Critical Studies in Education, 59(2), 187–203.

26.

Fehr

L. A.

Stamps

L. E.

McMahon

P. M.

(1981). Journal evaluations: Are they really evaluations of authors? The American Psychologist, 36(11), 1456–1457.

27.

Feingold

(1989). Assessment of journals in social science psychology. The American Psychologist, 44(6), 961–964.

28.

Ferrer-Sapena

Sánchez-Pérez

E. A.

Peset

González

L.-M.

Aleixandre-Benavent

(2016). The impact factor as a measuring tool of the prestige of the journals in research assessment in mathematics. Research Evaluation, 25(3), 306–314.

29.

Fister

Perc

(2016). Toward the discovery of citation cartels in citation networks. Frontiers of Physics, 4(49). https://doi.org/10.3389/fphy.2016.00049

30.

Giles

(2005). Internet encyclopaedias go head to head. Nature, 438, 900–901.

31.

Hegarty

Walton

(2012). The consequences of predicting scientific impact in psychology using journal impact factors. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 7(1), 72–78.

32.

Humphrey

Kiseleva

Schleicher

(2019). A time-series analysis of the scale of coercive journal self-citation and its effect on impact factors and journal rankings. European Accounting Review, 28(2), 335–369.

33.

James

(2016). WikiProject Medicine: Creating credibility in consumer health. Journal of Hospital Librarianship, 16(4), 344–351.

34.

Jemielniak

(2014). Common knowledge? An ethnography of Wikipedia. Stanford University Press.

35.

Jemielniak

(2016). Wikimedia movement governance: The limits of a-hierarchical organization. Journal of Organizational Change Management, 29(3), 361–378.

36.

Jemielniak

Aibar

(2016). Bridging the gap between Wikipedia and Academia. Journal of the Association for Information Science and Technology, 67(7), 1773–1776.

37.

Jemielniak

Greenwood

D. J.

(2015). Wake up or perish: Neo-Liberalism, the social sciences, and salvaging the public university. Cultural Studies↔ Critical Methodologies, 15(1), 72–82.

38.

Jemielniak

Masukume

Wilamowski

(2019). The most influential medical journals according to Wikipedia: Quantitative analysis. Journal of Medical Internet Research, 21(1), e11429.

39.

Jemielniak

Wilamowski

(2017). Cultural diversity of quality of information on Wikipedias. Journal of the Association for Information Science and Technology, 68(10), 2460–2470.

40.

Kaslow

N. J.

(2015). Translating psychological science to the public. The American Psychologist, 70(5), 361–371.

41.

Konieczny

(2016). Teaching with Wikipedia in a 21st-century classroom: Perceptions of Wikipedia and its educational benefits. Journal of the Association for Information Science and Technology, 67(7), 1523–1534.

42.

Konieczny

(2017). Decision making in the self-evolved collegiate court: Wikipedia’s Arbitration Committee and its implications for self-governance and judiciary in cyberspace. International Sociology: Journal of the International Sociological Association, 32(6), 755–774.

43.

Koulack

Keselman

H. J.

(1975). Ratings of psychology journals by members of the American Psychological Association. The American Psychologist, 30(11), 1049–1054.

44.

Kulczycki

(2019). Field patterns of scientometric indicators use for presenting research portfolio for assessment. Research Evaluation, 28(2), 169–181.

45.

Leydesdorff

Opthof

(2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science. American Society for Information Science, 61(11), 2365–2369.

46.

London

D. A.

Andelman

S. M.

Christiano

A. V.

Kim

J. H.

Hausman

M. R.

Kim

J. M.

(2019). Is Wikipedia a complete and accurate source for musculoskeletal anatomy? Surgical and Radiologic Anatomy: SRA. https://doi.org/10.1007/s00276-019-02280-1

47.

Macdonald

Kam

(2007). Aardvark et al.: Quality journals and gamesmanship in management studies. Journal of Information Science and Engineering, 33(6), 702–717.

48.

Martin-Sardesai

Irvine

Tooley

Guthrie

(2017). Government research evaluations and academic freedom: A UK and Australian comparison. Higher Education Research & Development, 36(2), 372–385.

49.

Mayor

(2010). Are scientists nearsighted gamblers? The misleading nature of impact factors. Frontiers in Psychology, 1 (215). https://doi.org/10.3389/fpsyg.2010.00215

50.

McPhetres

(2019). A perspective on the relevance and public reception of psychological science. Collabra: Psychology, 5(1). https://doi.org/10.1525/collabra.223

51.

Mesgari

Okoli

Mehdi

Nielsen

F. Å.

Lanamäki

(2015). “The sum of all human knowledge”: A systematic review of scholarly research on the content of Wikipedia. Journal of the Association for Information Science and Technology, 66(2), 219–245.

52.

Michelucci

Dickinson

J. L.

(2016). The power of crowds. Science, 351(6268), 32–33.

53.

Moustafa

(2015). The disaster of the impact factor. Science and Engineering Ethics, 21(1), 139–142.

54.

Moustafa

(2016). Aberration of the citation. Accountability in Research, 23(4), 230–244.

55.

Neff

M. W.

(2018). Publication incentives undermine the utility of science: Ecological research in Mexico. Science & Public Policy, 45(2), 191–201.

56.

Niederer

van Dijck

(2010). Wisdom of the crowd or technicity of content? Wikipedia as a sociotechnical system. New Media & Society, 12(8), 1368–1387.

57.

Nkomo

S. M.

(2009). The seductive power of academic journal rankings: Challenges of searching for the otherwise. Academy of Management Learning & Education, 8(1), 106–121.

58.

Oeberst

von der Beck

D Back

Cress

Nestler

(2018). Biases in the production and reception of collective knowledge: The case of hindsight bias in Wikipedia. Psychological Research, 82(5), 1010–1026.

59.

Okumus

Zhao

X. R.

Van Niekerk

Law

(2018). The importance of having a balanced rating index for ranking academic journals. Journal of Hospitality & Tourism Research, 42(7), 1170–1181.

60.

Osterloh

Frey

B. S.

(2015). Ranking games. Evaluation Review, 39(1), 102–129.

61.

Over

(1978). Journal ranking by citation analysis: Some inconsistencies. The American Psychologist, 33(8), 778–780.

62.

Özbilgin

M. F.

(2009). From journal rankings to making sense of the world. Academy of Management Learning & Education, 8(1), 113–121.

63.

Pagani

R. N.

Kovaleski

J. L.

Resende

L. M.

(2015). Methodi Ordinatio: A proposed methodology to select and rank relevant scientific papers encompassing the impact factor, number of citation, and year of publication. Scientometrics, 105(3), 2109–2135.

64.

Papakostidis

Giannoudis

P. V.

(2018). Impact factor and altmetrics: What is the Future? In Mauffrey

Scarlat

M. M.

(Eds.), Medical writing and research methodology for the orthopaedic surgeon (pp. 71–79). Springer.

65.

Paulus

F. M.

Cruz

Krach

(2018). The impact factor fallacy. In Frontiers in Psychology (Vol. 9). https://doi.org/10.3389/fpsyg.2018.01487

66.

Perez

Bar-Ilan

Cohen

Schreiber

(2019). The network of law reviews: Citation cartels, scientific communities, and journal rankings. The Modern Law Review, 82(2), 240–268.

67.

Rafols

Leydesdorff

O’Hare

Nightingale

Stirling

(2012). How journal rankings can suppress interdisciplinary research: A comparison between innovation studies and business & management. Research Policy, 41(7), 1262–1282.

68.

Reagle

J. M.

(2010). Good faith collaboration: The culture of Wikipedia. MIT Press.

69.

Reavley

N. J.

Mackinnon

A. J.

Morgan

A. J.

Alvarez-Jimenez

Hetrick

S. E.

Killackey

Nelson

Purcell

Yap

Jorm

A. F.

(2012). Quality of information sources about mental disorders: A comparison of Wikipedia with centrally controlled web and printed sources. Psychological Medicine, 48(8), 1753–1762.

70.

Rushforth

de Rijcke

(2015). Accounting for impact? The journal impact factor and the making of biomedical research in the Netherlands. Minerva, 53(2), 117–139.

71.

Rushton

J. P.

Roediger

H. L.

(1978). An evaluation of 80 psychology journals based on the science citation index. The American Psychologist, 33(5), 520.

72.

Schweitzer

N. J.

(2008). Wikipedia and psychology: Coverage of concepts and its use by undergraduate students. Teaching of Psychology, 35(2), 81–85.

73.

Seglen

P. O.

(1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498–502.

74.

Shane-Simpson

Che

Brooks

P. J.

(2016). Giving psychology away: Implementation of Wikipedia editing in an introductory human development course. Psychology Learning & Teaching, 15(3), 268–293.

75.

Sorokowski

Kulczycki

Sorokowska

Pisanski

(2017). Predatory journals recruit fake editor. Nature, 543(7646), 481–483.

76.

Stamps

L. E.

Fehr

L. A.

(1980). Journal ratings revisited: A closer scrutiny of impact factors. The American Psychologist, 35(11), 1048–1049.

77.

Strathern

(1997). “Improving ratings”: Audit in the British university system. European Review, 5(3), 305–321.

78.

Stroebe

Strack

(2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 9(1), 59–71.

79.

Surowiecki

(2004). The wisdom of crowds. Anchor Books.

80.

Tadajewski

(2016). Academic labour, journal ranking lists and the politics of knowledge production in marketing. Journal of Marketing Management, 32(1-2), 1–18.

81.

Teplitskiy

Duede

(2017). Amplifying the impact of open access: Wikipedia and the diffusion of science. Journal of the Association for Information Science and Technology, 68, 2116–2127.

82.

Uhlmann

E. L.

Ebersole

C. R.

Chartier

C. R.

Errington

T. M.

Kidwell

M. C.

Lai

C. K.

McCarthy

R. J.

Riegelman

Silberzahn

Nosek

B. A.

(2019). Scientific Utopia III: Crowdsourcing science. Perspectives on Psychological Science: A Journal of the Association for Psychological Science. https://doi.org/10.1177/1745691619850561

83.

Vanclay

J. K.

(2012). Impact factor: Outdated artefact or stepping-stone to journal certification? Scientometrics, 92(2), 211–238.

84.

Vogl

Scherndl

Kühberger

(2018). #Psychology: A bibliometric analysis of psychological literature in the online media. Scientometrics, 115(3), 1253–1269.

85.

White

M. J.

White

K. G.

(1977). Citation analysis of psychology journals. The American Psychologist, 32(5), 301.

86.

Wikipedia. (2019). Wikipedia: Size comparisons. In Wikipedia, the Free Encyclopedia. https://perma.cc/K4BT-QJYD