Abstract
Most definitions of “open science” focus quite narrowly on the means of knowledge production. I noted this in my previous Beyond Open Science editorial, that open science is chiefly seen as a way of generating and sharing knowledge. For example, Fecher and Friesike (2014)’s influential model of the “five schools of thought” in open science focuses chiefly on aims for and assumptions about individual academic outputs. For the Democratic School, access to knowledge is unequally distributed, so knowledge outputs, like papers and code, ought to be permissively licensed and shared freely. For the Pragmatic school, open code makes scientists more efficient at producing knowledge. The closely aligned Infrastructure school of open science turns this attention to institutions, services, and technology responsible for sharing these outputs, while the Measurement school seeks to improve methods to understand how science itself can be valued. Only the Public school, which focuses on the intelligibility and impact of science to members of the general population, could be seen as naturally extending beyond the scope of a single publication to the wider role of science in society. 1 Thus, open science often refers to a collection of researcher practices that exist fundamentally at the publication-level.
Replication, a core concept in open science, is similarly situated. Most definitions of replication have an episodic understanding of science. That is, replications are events: individual attempts at studying (or re-studying) a process using a consistent set of data, theory, and methods. In this understanding, a replicable finding requires a constellation of replication events, each self-consciously building on and critiquing previous attempts. Indeed, replication is often a core mode of a discipline’s process for self-critique. For us to be aware of progress, we need points of precise comparison to demonstrate how our understanding has grown. In this view, replication is when your community checks its understanding. It seems likely, then, that replication events will continue to be at the core of whatever open science sits beyond our current horizon, since replication helps us build an understanding of how we have improved upon past work.
A popular definition of replication sees this process as just part of how science is normally done, and that talking about “replication” explicitly is too self-serious. Instead of self-consciouly building on and critiquing previous work, we can “let a hundred flowers bloom,” so to speak. What truly grounds replicable knowledge is our ability to synthesize a consensus from similar studies approaching similar topics in similar ways. If we think a cluster of these papers has similar results, then we can interpret them as mutually supporting the replicability of their results. That is, replication is when your community can't stop talking about something. In this way, no one has to re-examine anyone else’s work in a specific, detailed, or self-conscious fashion. Rather, a judicious literature reviewer can, in hindsight, curate a cluster of papers that all seem to engage with the same topic and tend to find the same thing.
From within a community of practice, it is straightforward to view new work as if it were always a replication of things that have come before. After all, we all stand on the shoulders of giants yet build ever upwards towards a better understanding of our concepts and processes. Rather than self-correcting, science is self-improving—we can’t know, of course, once this improvement is “done,” and thus it will never be finished improving in this fashion. I call this whig replication. This is by analogy to “whig history,” an approach to history that sees the present as a more moral, ethical, or enlightened state than the past because it benefits from all of the ingenious labor saving innovations, improvements to civil rights and political systems, and new or better ways to live. Whig replication is at once progressive in its belief about the ability of science to continually improve our understanding, but is conservative in its actions to protect community knowledge from direct contestation and re-examination.
This whiggish idea of replication is also a reasonable response to the “replication crisis,” re-focusing it as a crisis of scientific meaning—what really matters to us is the meaning that we can make from scientific outputs, rather than the meaning that our scientific outputs purport to tell others in their results and conclusion sections. In either case, replication still helps us build a useful understanding from the outputs of a flawed political economy of research. Indeed, it seems that the “strain on scientific publishing” (Hanson et al., 2024) makes it increasingly difficult to imagine any one paper attempting seriously to be in conversation with any other when the room itself is so loud, inchoate, and vast. Further, as new ideas expand, it gets harder to find common ground on which to assess their quality. As Carlile (2004, p. 557) puts it, common knowledge is useful in specialist communities when assessing the quality or efficacy of a given design. However, this stock of common knowledge reduces when faced with new ideas or techniques, and it becomes harder to judge quality as a consequence.
Thus, whether as an event or something else, replication plays an increasingly important role in making meaning within our academic communities, especially when publication of a paper loses its ability to signal quality. As Fiala and Diamandis (2019) write, “the importance of publishing in high-impact journals (many of which are closed access, and therefore subscription journals) is progressively decreasing due to the rise of open access journals and other publication options” (p. 2). Concern about the increasing speed and quality of academic output is not new of course, in the sense that it has long been straightforward to “generate nonsense faster than ever before” (Rhind, 1985). But perhaps the sheer scale of the growth in publication speed and volume poses fresh issues to human understanding. For example, Hanson et al. note that, for some publishers, regular papers are vastly outnumbered by papers in special issues. How special can these issues really be?
Alongside folk replication practices, one structure we can use to help curate papers in our discipline is to start (or contribute to) overlay journals. Overlay journals generally curate articles from open access preprint repositories instead of offering their own primary path to publication. The paper “lives” on the preprint repository, while the journal offers visibility, review, and context to the preprint. Like a peer-reviewed “best things I’ve read this week” newsletter, overlay journals focus more clearly on a journal’s role to shepherd a community’s interests and discussions while they jettison the conventional obligation a journal had to produce the printed/typeset artefact itself. And, while traditional journals derive value in part from the compounding of citations to their publications over time, overlay journals are beholden to no such founder effect, since their papers exist independently of the overlay journal as citable preprint artefacts before curation. Thus, alongside replication, overlay journals might serve as a way for us to come together and make meaning from the massive body of academic work.
Another interesting aspect of the overlay journal is that part of its value comes from puncturing our academic bubble. Ideally, overlay journals surface work that is useful or interesting, but also work that may not have otherwise surfaced to their readers through social networks or algorithmically mediated newsfeeds or recommendations. They do not necessarily rely on the literature submitted to them directly. The fact that papers exist independently of the overlay journal makes the artisanal value added by curation even clearer. In this sense, overlay journals could serve a similar function to a local record shop—instead of seeing the newest cool paper on LinkedIn or Bluesky, we might instead learn about the hip underground publications from the aficionados at the hole in the wall around the corner. 2 Indeed, local independent record shops are making a comeback 3 (Evitts and Trenholm, 2024), so I think we’re seeking exactly this kind of taste making—an need for an alternative academic media.
This also empowers new ideas about what a journal itself can do, since curation is not limited to the production priorities of traditional journals. For example, the Ladies of Landsat’s (2023) Manuscript Monday featured “recently published, cutting-edge research led by under-represented scientists in the field of remote sensing” from 2019 to 2023. One can easily imagine an overlay journal operating in a similar fashion: highlight a diverse set of scholars doing cool work in novel or stimulating ways, unburdened by the contexts in which those papers were published, hosted, or archived. To be clear, the existence of an alternative scientific media certainly does not relieve traditional academic media from their responsibility to improve diversity, equity, and inclusion of themselves or the communities they steward. Instead, overlay journals offer us a way to “do it yourself.” Importantly, this DIY route is actually legible to the traditional processes of tenure, promotion, and research assessment in a way that a zine or a record shop is not.
Where this analogy breaks down, of course, is that record shops must buy and hold inventory, and they must pay rent. However, there is basically no barrier to entry for an overlay journal, and its maintenance costs only involve a subset of those for traditional journals. Thus, while overlay journals may serve as a very useful avenue for making meaning from an increasingly large volume of publications, it seems likely that overlay journals would themselves compound the rate of publication growth. If it’s enough to post a preprint and get “Paper of the Month” at your favorite overlay journal, who would wait 9 months for Reviewer two to majorly revise your tables as charts?
Perhaps this is where replication as a meaning-making practice still can help. An overlay journal could curate collections of papers that “hang together” in a narrative sense alongside replication events. Ultimately, though, replication itself needs to help us identify both replicable and non-replicable knowledge. Whig replication cannot do this. A literature review can talk about the genealogy of a concept but cannot decisively determine whether two studies replicate if their designs are not comparable. Instead, in whig replication, we simply choose not to engage work that we think is not replicable. That is, non-replication becomes something we don’t want to talk about, work that is subtly different from what is common within our community is simply ignored. Papers that don’t agree with our body of knowledge are talking about something different or doing something wrong. Contrast this with the explicit replication failures explored in city science over 50 years ago, which still has value for understanding work today (Oshan, 2022). This understanding can only be built if we design situations where replication can fail, not avoid discussing difference. In this sense, it seems unlikely that whig replication or curation generally can support knowledge that extends beyond the communities and times in which it is directly embedded and produced.
Whig replication may be useful to grow or consolidate a scholarly community, but we need actual intentional replication events, too, to bring our perspectives together and test our common understandings. Yet, curation will only become more important as the literature continues to grow, so overlay journals and other alternative academic media offer new and useful ways to get directly involved. Thus, replication in either sense will continue to create value for city science if we are diligent (and deliberate) about it.