Abstract
Understanding more about variations in peer review is essential to ensure that editors and reviewers harness it effectively in existing and new formats, including for mega-journals and when published online. This article analyzes open reviews from the MDPI suite of journals to identify commonalities and differences from a simplistic quantitative perspective, focusing on reviewer anonymity, review length, and review outcomes. The sample contained 45,385 first round open reviews from published standard journal articles in 288 MDPI journals classified into one or more Scopus disciplinary areas (Health Sciences; Life Sciences; Physical Sciences; Social Sciences). The eight main findings include substantial differences between journals and disciplines in review lengths, reviewer anonymity, review outcomes, and the use of attachments. In particular, Physical Sciences journal reviews tended to be stricter and were more likely to be anonymous. Life Sciences and Social Sciences reviews were the longest overall. Signed reviews tend to be 15% longer (perhaps to be more careful or polite) but gave similar decisions to anonymous reviews. Finally, reviews with major revision outcomes tended to be 68% longer than reviews with for minor revision outcomes, except in a few journals. In conclusion, signing reviews does not seem to threaten the validity of peer review outcomes and authors, editors, and reviewers of multidisciplinary articles should be aware of substantial field differences in what constitutes an appropriate review.
Introduction
Peer review is the core quality control mechanism to increase the validity of formally published academic studies. Its purpose is to identify and correct errors and filter out invalid or redundant studies. It may also filter out studies with less important valid contributions, although they may still be published in lower prestige journals (Casnici et al., 2017) or open access repositories (Hoy, 2020). Despite widespread academic belief in its value, peer review is imperfect, with many different sources of bias (Lee et al., 2013). High-profile instances of failure typically lead to retractions (e.g. Ledford and Van Noorden, 2020) and initially rejected important papers are a different type of failure (Campanario, 1996). It is therefore important to investigate peer review outcomes, when possible, including differences between disciplines, journals, and reviewer anonymity status. It is also particularly important to investigate peer review when it is applied in new types of journal, or when it is applied in different ways (e.g. open and non-anonymous reviews).
The emergence of new types of journals, including multidisciplinary mega-journals (e.g. Plos One) and multidisciplinary data/methods/software journals (e.g. Data in Brief, JoVE, Journal of Open Research Software) has implications for peer review. Some of these journals do not have importance thresholds for research, altering the core reviewing criteria. For example, Plos One states, “In keeping with our mission to publish all valid research, we consider negative and null results” (https://journals.plos.org/plosone/s/criteria-for-publication). The multidisciplinary nature of these new types of journals also potentially changes the nature of reviewing because the publishing criteria must necessarily be general enough for all disciplines and both editors and reviewers seem more likely to deal with articles from outside their home disciplines. These developments make disciplinary variations in peer review more important.
A second current trend in peer review is the increasing use of open reviews, where reports are published alongside published articles, sometimes signed by the reviewers (Ross-Hellauer et al., 2017). This may be thought a public good, as part of a drive to increase openness in science (Tennant, 2018). Open reviews may also safeguard against reviewer bias, although double-blind peer review has other safeguards against bias (Haffar et al., 2019; Lee and Moher, 2017). Open peer review might induce biases if reviewers are unwilling to publicly criticize anyone or people that they know (e.g. Thelwall et al., 2021). It is not clear whether overall review quality is impacted by these issues, however (Bruce et al., 2016), and double-blind peer review seems to be preferred by most researchers (Mulligan et al., 2013; Rodríguez-Bravo et al., 2017). Only 6% of journals in a recent multidisciplinary survey (n = 322 editors) used a form of open peer review (Hamilton et al., 2020). Nevertheless, at least 536 journals had published at least one open peer review report by 2019 (Wolfram et al., 2020). One publisher, MDPI (Multidisciplinary Digital Publishing Institute), accounted for 33% of the journals and the main disciplinary areas were health sciences, medical sciences, and natural sciences. About half of the journals had mandatory OA reports, with the rest mainly being optional at the request of the author. Reviewers could usually (56%) opt to reveal their identities. More information about potential biases in aspects of open peer review in different contexts is needed to help journals and authors to decide on whether to adopt it, however.
Little is known about disciplinary differences in peer review because reports are usually not published, with the exceptions noted above. Peer review has often been investigated on a small scale or analytically rather than for disciplinary differences, with a few exceptions. One indirect analysis examined 3500 author evaluations of the reviewing process of journals on the SciRev website, finding small disciplinary differences in authors’ evaluations of review quality and the difficulty of revisions requested by reviewers (Huisman and Smits, 2017). For example, authors thought that mathematics asked for the easiest revisions and economics and business asked for the hardest revisions. A literature survey of reported acceptance rates also found disciplinary differences, for example with Biomedicine being less selective than social sciences (Bjórk, 2019). An investigation into reviewer statistics reported in the Publons website found national, gender, and disciplinary differences in patterns of reviewing (Xu et al., 2019). A more detailed analysis of disciplinary differences would be useful to help editors and reviewers that cross disciplinary boundaries, however.
This article analyzes disciplinary differences in the properties of the open peer review reports of journals from the MDPI publisher. This set was chosen as a large multidisciplinary source of open journal article reviews (Wolfram et al., 2020), allowing a broad analysis of differences. A disadvantage of MDPI compared to a few other sources, such as F1000Research, is that it does not publish the reviews for rejected articles. Another disadvantage is that open peer review is optional for MDPI authors, so the review dataset is incomplete. MDPI is an open access, pay-to-publish publisher, with multidisciplinary journals offering relatively high acceptance rates combined with fast publishing speeds (Petrou, 2020). Many of its journals are indexed in the Web of Science (Oviedo-García, 2021) and Scopus (see below). It primarily works through special issues, the success of which have driven its dramatic growth (Crosetto, 2021). MDPI authors can choose to have their peer review reports published, but reviewers can opt to remain anonymous. MDPI first trialed this process in 2014 with its journal Life (Rittman, 2014), then expanded it to 14 more before applying it to all journals from October 2018 (Rittman, 2018). The stated objectives were to increase transparency, credit reviewers (Publons is an anonymous alternative: Ortega, 2019), and improve the quality of comments.
This article addresses one specific research question about disciplinary differences and one about peer review anonymity. The key variables of review anonymity and outcome are investigated as central to peer review. Review length is also assessed as a simple proxy for review quality. The presence of attachments in reviews is included for convenience as a minor point of interest and potential difference.
RQ1 Are there any differences between journals and disciplines in the length, outcome, anonymity (signed and unsigned), and attachments for MDPI open peer reviews?
RQ2 Are there any length and outcome differences between signed and unsigned MDPI open reviews?
Methods
The research design was to download a large set of first round open peer review reports from MDPI journals, statistically evaluating the results as appropriate to answer the research questions.
Data
The MDPI website was initially processed by the free software Webometric Analyst (https://lexiurl.wlv.ac.uk) to extract all articles from its journals at a rate of one URL per second 29–30 July 2021. The sitemap XML files in the website were downloaded and the URLs in them extracted to a list of article URLs (the Sitemap XML: Extract all URLs option in Webometric Analyst’s Services menu) and this list was used to download these articles (the Download binary or text URLs option in Webometric Analyst’s Services menu), although they had last been updated on 25 November 2020, so miss almost all of 2021 (some 2021 papers had been published in 2020) and some of 2020, so these years are incomplete. There is a separate sitemap for each journal. This produced 43,935 webpages for 293 journals, each representing a single journal article. Data checking by comparing the number of articles found by the crawler with the number of articles available on the MDPI website for some journals revealed large gaps in the coverage, so MDPI seems to have a policy of putting a fraction of its articles into its sitemap.
In response to the sitemap gaps, a list of MDPI journal homepages was made from the journal list inside the website (https://www.mdpi.com/about/journals/sci) in August 2021 and a custom crawler built and added to the free software Webometric Analyst (http://lexiurl.wlv.ac.uk, Services menu, MDPI submenu) to crawl the articles from each journal. The custom crawler was coded to recognize the HTML structure of MDPI journal home pages, parsing them to find all volumes for each journal. It repeated this for volume home pages and issue home pages to get a list of articles for each issue. The results of this crawler are more extensive than from the sitemap but still incomplete for journals that have large numbers of articles per issue. This is because some of the journals use JavaScript Ajax to add extra articles in a user’s web browser. This limitation was accepted as unproblematic because it concerned only large individual issues of journals and therefore should not result in biased samples unless the order of articles within an issue interferes with the results. This is possible since many articles are in special issues where the first published would presumably be the first written or the fastest reviewed, but this seems unlikely to have a substantial impact on the results.
Documents published in MDPI journals are assigned a type in issue web pages, such as “Article,” “Review,” “Entry,” “Letter,” “Reply,” “Communication.” Only files of type journal article were downloaded from issues, as flagged in the HTML, and this resulted in some journals reporting no records (e.g. Encyclopedia, in which all publications are labeled as type “Entry”: https://www.mdpi.com/journal/encyclopedia). A total of 159,153 article webpages were downloaded from 345 journals (out of 361 crawled) in August 2021. They were used instead of, rather than in addition to, the sitemap pages to give a consistent dataset.
The downloaded article files were parsed by a program added to Webometric Analyst (Services menu, MDPI submenu, MDPI saved article html: Extract information and review URLs) to identify whether there was a hypertext link in the article HTML to a page hosting the review files for the articles (i.e. a hyperlink to the article URL with “/review_report” at the end). This link was present in 11% of the pages and took the form of a button near the top of the article.
The 17,753 review webpages linked to by the article pages were downloaded in August 2021 using the URLs extracted as described above. The reviews were extracted from the relevant sections of the HTML of each review webpage as well as the publication date, journal, and reviewer names. For example, reviewer reports were preceded by standard HTML, such as “Reviewer 1 Report,” that could be identified by the program. MDPI reviews are described as occurring in “rounds,” where Round 1 is the initial reviewing, to which the authors reply, then the reviewers may be sent the revision and submit a subsequent review. Round 2 contains these follow-up reviews usually from the same reviewers, plus author responses. There can be additional rounds of reviews. For this study, only the Round 1 reviews were used because subsequent rounds of reviews relate to the initial rounds and are therefore dependent. For example, one Round 2 review simply stated, “My past review comments have not been addressed.” These steps were completed again using Webometric Analyst (MDPI submenu).
Reviews data processing
The text in the review web pages was processed to extract information about the review decision, the presence of attachments, and the length of the review.
Outcome: A Round 1 review was judged to have an outcome of accept or minor revision (henceforth: minor revision) if there was no Round 2 review from the same reviewer, otherwise it was judged to be a reject or major revision outcome (henceforth: major revision). This is an oversimplification because a reviewer may have ignored a request for re-review after a reject or major revisions or an editor may have overridden their original decision. In at least one case (although not a Round 1 review), this seems to have happened. This heuristic was necessary because the report files did not include reviewer decisions.
Presence of attachments: Reviews sometimes contained a HTML link to PDF or Microsoft Word attachments with all or part of the review. These links were easy to identify in the underlying HTML.
Review length: The length of a review was captured as the number of characters contained in it, after removing all HTML tags. Counting characters rather than words adds weight to longer words, which are typically more complex (Lewis and Frank, 2016), and to long mathematical expressions. The shortest Round 1 review was, “Good” and the longest had 49,454 characters, fitting on 13 A4 pages single spaced in Times New Roman font (Reviewer 2: https://www.mdpi.com/2076-328X/9/12/142/review_report). Reviews with attachments were excluded from the length analyses because it was not possible to assess their length in a reasonable way. Whilst some were documents containing just the review, they often included metadata (e.g. article titles) that added to their length. Some were also annotated copies of the submitted journal article, or copies of the text of the review in HTML in the website. In all these cases, the length of the PDF file or Word document would overestimate the length of the review. Thus, length information for reviews with attachments was ignored as unreliable.
Disciplinary differences: These were examined using an independent broad disciplinary categorization: the four top level categories of Scopus (August 2021 source list Excel file downloaded 14 September 2021 from here: https://www-elsevier-com-s.web.bisu.edu.cn/solutions/scopus/how-scopus-works/content), as compiled by a committee at Elsevier, supported by citation data. The Scopus system was used because it covers more journals than the Web of Science (Martín-Martín et al., 2021) and the broadest categories available were needed to allow large enough groups for journal-level statistical analyses. Journals without these categories were given the categories of other journals with similar names (only journals with at least 30 review reports, since the others were not analyzed). Three journals could not be given a reasonable category in this way and were excluded: J and Sci (cover all academic research); and Vehicles (no similar Scopus journal). Within the 25 journals with the most reviews, for example, 10 were given multiple categories and no journal only had the Social Science category (Table 1).
Scopus top-level categories for 25 MDPI journals. The complete set of categories is online (https://doi.org/10.6084/m9.figshare.16677091).
Category taken from Scopus for Nihon Rinsho. Japanese Journal of Clinical Medicine.
Category taken from Scopus for Revista Brasileira de Medicina.
Statistical analysis
Relationships other than length were in the form of tables and were examined with chi square tests, with a Yates correction when necessary. Secondary tests are reported for changes in properties over time to check for evolutions that could have a second-order impact on the results. Length-based comparisons were made with geometric means rather than arithmetic means due to skewing (a small number of long reviews). Confidence intervals (using the normal distribution formula on the logged raw data produced for the geometric mean) were calculated for comparisons between individual journals.
Statistical tests for discipline used each journal average as a data point rather than each individual article to avoid giving higher weights to larger journals. Only journals with at least 30 articles were included, however, so ensure that these averages were reasonably precise.
Linear regression was used to assess the contribution of each discipline to the likelihood that a review is signed, has an attachment, or has a major revision outcome, as well as its length. The four independent variables were the four disciplines, with a value of 0 if the journal was not within the discipline and 1/n if the journal was in the discipline, where n is the number of disciplines that the journal was classified in. For example, a review for a journal classified as both Physical Sciences and Social Sciences would have dependant variable values of: Physical Sciences: ½; Social Sciences: ½; Health Sciences: 0; Life Sciences: 0. This assumes that each discipline contributes equally to the reviewed article and that the contributions add linearly. Since the values of the four independent variables sum to 1 for each review, only three could be included in the regression (collinearity). Since none of the variables formed an obvious baseline, the regressions were run four times, excluding each discipline in turn. The left out variable in each case formed the baseline discipline.
A robust version of linear regression, as implemented in the R package robustbase, through the function lmrobDetMM (Mächler and Ruckstuhl, 2006), was used. This implements a weighted version of least squares that is more robust to high leverage outliers (Maronna et al., 2019; Yohai, 1987), as present in some regressions. Since the cut-off of 30 reviews (i.e. minimum number of reviews per journal) was arbitrary and there are outliers in the data, the analyses were repeated with cut-offs of 50 and 100. Code, journal-level data and outputs are in the Supplemental Material: https://doi.org/10.6084/m9.figshare.16677091.
Results
The results are reported first in terms of trends over time for first round open reviews, then for the properties of all first-round reviews, and finally for review lengths for first round open reviews without attachments. Although identifying changes over time is not relevant to the main goals, any substantial changes would point to the need for extra care interpreting relevant statistical analyses. The main findings are labeled with letters (F1, F2, etc.) to aid navigation.
All round 1 open reviews: Temporal trend checks
The proportion of signed round 1 reviews differed statistically significantly by year (Table 2, chi square 34.0, p = 0.000, excluding 2015), with signing being more prevalent in 2015–2017 than in 2018–2020. Low proportions of signed reviews align with a belief that signing reviews should not be compulsory for reviewers (Ross-Hellauer et al., 2017).
The proportion of anonymous or signed first round open reviews, by year.
The proportion of reviews with a major revision or reject decision did not differ statistically significantly between years (Table 3, chi square 6.7, p = 0.346, excluding 2015), at 40% overall.
The proportion of first round open reviews with major revision or reject decisions, by year.
All round 1 open reviews: Relationships between decisions, signing, and attachments
Relationships between signed open reviews, reviews with attachments (see Appendix Table A1), and reviews with major revision/reject decisions were tested for with three chi square tests (one for each pair). There was a statistically significant relationship between review outcome and attachments, with major revision reviews being more likely to have an attachment (8.7%) than were minor revision reviews (7.7%) (chi square 14.9, p = 0.000, n = 4385), which is unsurprising since an accept or minor revisions decision may have little to report.
There was a statistically significant relationship between reviewer anonymity and attachments, with signed reviews being more likely to have an attachment (11.3%) than were anonymous reviews (7.7%) (chi square 75.8, p = 0.000, n = 4385). This may reflect small intersections between disciplinary cultures for reviewing styles or document formats (e.g. PDF) and open reviewing.
There was insufficient statistical evidence of a relationship between reviewer anonymity and review outcome (F1), with signed reviews being almost exactly as likely to have a minor revisions outcome (39.8%) as were anonymous reviews (40.0%) (chi square 0.1, p = 0.772, n = 4385). It is not known whether the finding would be true for rejected articles, however, since the reviews for these are not published.
All round 1 open reviews: Differences between journals
Out of the 221 MDPI journals with at least one open peer review article in the dataset, those with at least 100 were selected to compare the proportions with signed, major/reject decisions, and attachments (Figure 1). Differences between these journals in outcomes, attachments, and reviewer anonymity were statistically significant (chi square, p = 0.000) (F2). Moreover, large differences are evident for each facet. The extreme cases show the range: Nanomaterials (48%) had more than twice as many major revision decisions as JoF (fungi, 23%); over a fifth of Water (22%) open reviews had an attachment but none from Brain Sciences (0%) did; Nearly seven times as many Healthcare open reviews were signed (23%) than were signed for Nanomaterials (3%).
The proportion of signed open reviews, reviews with attachments, and reviews with major revision/reject decisions for the 25 MDPI journals with the most open reviews (min: 472 open reviews).
Linear regressions were conducted on all journals with at least 30 reviews (with the proportions as dependant variables) to test for disciplinary differences in the three proportions, with the following results (full details in the Supplemental Material: https://doi.org/10.6084/m9.figshare.16677091).
Major revisions: Physical Sciences reviews were 0.04–0.05 more likely to request major revisions than the other fields (F3) (Table 4).
Attachments: Physical Sciences and Social Sciences reviews were 0.027 more likely to have attachments than Life Sciences and Health Sciences reviews (F4) (Table 5).
Signed reviews: Physical Sciences reviews were the least likely to be signed; Social Sciences and Health Sciences reviews were the most likely to be signed (F5) (Table 6).
Linear regression coefficients for four regressions for the disciplinary contributions toward the proportion of major revision outcomes for a journal.
Significant at p < 0.05.
Each regression has a different discipline omitted, forming the baseline. For MDPI journals with at least 30 reviews. The coefficients in the table represent the increased probability of major revisions outcomes in the variable field (column heading) compared to the baseline field (row heading). The coefficients in square brackets instead represent the probability for the baseline field.
Linear regression coefficients for four regressions for the disciplinary contributions toward the proportion of attachments in a journal’s reviews.
Significant at p < 0.01. ***Significant at p < 0.001.
Each regression has a different discipline omitted, forming the baseline. For MDPI journals with at least 30 reviews. The coefficients in the table represent the increased probability of attachments in the variable field (column heading) compared to the baseline field (row heading). The coefficients in square brackets instead represent the probability for the baseline field.
Linear regression coefficients for four regressions for the disciplinary contributions toward the proportion of signed reviews for a journal.
Significant at p < 0.05. **Significant at p < 0.01. ***Significant at p < 0.001
Each regression has a different discipline omitted, forming the baseline. For MDPI journals with at least 30 reviews. The coefficients in the table represent the increased probability of signed reviews in the variable field (column heading) compared to the baseline field (row heading). The coefficients in square brackets instead represent the probability for the baseline field.
The results point to Physical Sciences being consistently different for open peer review, with Social Sciences being partly different.
Round 1 open reviews without attachments: Differences between journals
Excluding reviews with attachments, average open review lengths differed substantially between journals (Figure 2) (F2 continued). Within the 25 journals with the most reviews, the biggest difference was that Forests reviews were 67% longer than Electronics reviews.
Average (geometric mean) first review length (characters) for the 25 MDPI journals with the most open reviews, excluding all reviews with attachments (min: 472 open reviews without attachments). Gray: Physical Sciences; Green: Life Sciences; Orange: Health Sciences; Blue: multiple.
Length differences are at least partly journal-specific rather than disciplinary because review lengths differ between journals with similar scopes, as the following examples illustrate.
The Journal of Clinical Medicine has much longer reviews than the related journals Medicina and Cancers. Medicina has been published by the Lithuanian University of Health Sciences since 1920, rather than created by MDPI, so may bring existing reviewing traditions and reviewer pools to MDPI. This may explain the difference. The narrower scope of Cancers may explain its length difference.
Forests has longer reviews than Plants, despite similar life sciences backgrounds.
It is possible to identify some broad disciplinary difference themes in the results, although it is difficult to draw strong conclusions since each journal has unique coverage. The four Scopus main categories (Health Sciences, Life Sciences, Physical Sciences, Social Sciences) that the journals have been assigned to (Table 1) were used for this. Figure 2 suggests that there is a slight tendency for Physical Sciences reviews to be shorter (including the two journals with the shortest reviews) and a stronger tendency for Life Sciences reviews to be longer (including the three journals with the longest reviews). Regressions for length against discipline for journals with at least 30 articles without attachments (data for articles without attachments was excluded), found evidence that Life Sciences and Social Sciences journals tend to have longer reviews than Physical Sciences and Health Sciences journals (up to 239 characters (19%) longer than the base length of 1265 characters) (F6) (Table 7).
Linear regression coefficients for four regressions for the disciplinary contributions toward geometric mean review length for a journal, excluding reviews with attachments.
Significant at p < 0.05. **Significant at p < 0.01. ***Significant at p < 0.001.
Each regression has a different discipline omitted, forming the baseline. For MDPI journals with at least 30 reviews without attachments. The coefficient represents the number of extra characters for average review lengths in the variable field (column heading) compared to the baseline field (row heading). The coefficients in square brackets instead represent the average length for the baseline field.
Excluding reviews with attachments, reviews with a major revision outcome tend to be longer than reviews with a minor revision outcome in all the largest journals (Figure 3) (F7). For journals with at least 30 reviews without attachments, the median minor revisions length was 1053 and the median major revision length was 69% longer at 1776 characters. In the complete set of journals analyzed, 260 had a greater average length for major revision reviews and 18 had longer average minor revisions reviews, out of the 278 journals with at least one review of each type (excluding reviews with attachments). Considering the journals with the most reviews, the largest exception was Diseases (23 major: 1090 characters; 37 minor 1177 characters), followed by Clocks & Sleep (29 major: 1462 characters; 23 minor 1708 characters) and Antibodies (18 major: 1044 characters; 28 minor 1060 characters). The remaining exceptions had fewer than 30 reviews. Of these three, the length difference is largest in Clocks & Sleep. Examining short major revision reviews in this journal suggests that occasionally reviews requesting apparently minor changes are flagged as requiring a second review (e.g. review 1 of https://doi.org/10.3390/clockssleep3020021 and review 2 of https://doi.org/10.3390/clockssleep3030024). This suggests a policy of requiring reviewers to re-check even after requesting minor changes in some cases. No systematic reason could be found for short major revision reviews or long minor revisions reviews, however.
Average (geometric mean) first review length (characters) by revision type for the 25 MDPI journals with the most open reviews, excluding all reviews with attachments (min: 472 open reviews without attachments).
A longer review for major revisions, as found overall and for nearly all journals, seems logical since a major revision is necessarily more substantial than a minor revision and presumably needs more words to explain in detail. This shows that the option to write a short major revision review, such as by briefly saying that something is flawed and then not reviewing the rest of the manuscript, would be the exception in all fields.
Excluding reviews with attachments, signed reviews tend to be longer in 80% (20) of the 25 journals with the most reviews, although the differences are often small (Figure 4). For the complete set of journals, 169 have longer signed reviews and 81 have longer anonymous reviews out of the 250 journals with both. For journals with at least 30 reviews without attachments, the median average anonymous review length was 1290 characters, and the median average signed review length was 15% longer at 1486 (F8). Slightly longer signed reviews might be expected if identified reviewers are more polite or explain points in greater detail.
Average (geometric mean) first review length (characters) by reviewer anonymity for the 25 MDPI journals with the most open reviews, excluding all reviews with attachments (min: 472 open reviews without attachments).
Discussion
The results have a wide range of limitations that make all the conclusions tentative rather than strong, and will benefit from triangulation in the future from new sources of reviews, as they emerge. First, all reviews are from one publisher, MDPI (Crosetto, 2021), which has a distinctive publishing model (gold open access, optional open reviews for all journals, many special issues with guest editors, fast reviewing, relatively large multidisciplinary journals), so the findings may well not apply to other publishing models. The results exclude rejected articles that might have had open reviews, if published (in contrast to some, such as F1000Research), biasing the review samples. Statistically, the data includes multiple reviews for same article, so the data points (individual reviews) are not independent, although referees historically have had a low or moderate degree of agreement (Baethge et al., 2013; Munley et al., 1988). The journal-level analyses may also suffer from imperfect independence due to publisher reputation in specific areas. For example, if MDPI has a higher reputation in one area then reviewers for journals from that area may tend to be more careful or critical. The assumption that it is reasonable to class a review as recommending a “major revision” if the reviewer posted a subsequent review is also a simplification for some or all the journals because a perfect answer to a major revision decision might not need a follow up minor revision. Thus, it is likely that some major revision decisions are included in this article in the minor revisions decisions. This would not invalidate the results but instead reduce the difference between the two sets by including some of one set in the other. Finally, some of the results are potentially applicable to closed peer review (e.g. disciplinary differences in article length) and it is not clear whether such extrapolations are reasonable, given the lack of evidence about closed peer review.
The main findings are discussed below and compared to what is already known from previously-published research, when relevant. Findings F1 and F8 address the second research question (possible influence of anonymity) and the remaining findings address the first research question (disciplinary and journal differences).
F1: The reviewer decision (major revision or reject vs accept or minor revisions) does not associate with their anonymity (i.e. whether the reviews are signed). This is perhaps surprising given that reviewers risk making an enemy with more negative decisions, and this may have repercussions if their own work is reviewed by one of the authors in future. On the other hand, there is also a risk of reputational damage from minor revisions or accept decisions if the article is published with flaws, or after flaws pointed out by other reviewers were corrected, because peers will know the identity of the reviewer that missed them. Since most reviewers prefer to remain anonymous (88.7%), the consensus seems to be that neither risk outweighs the possible credit for the work of reviewing or gratitude from the authors after their article is accepted. Thus, the results suggest that either the two risks balance out and therefore do not influence reviewers or that the minority opting for open review do not consider the risks, perhaps because they have ideological reasons for supporting open peer review in principle.
F2: There are substantial differences between journals in the proportions of reviews with attachments, major revision decisions, and reviewer anonymity, as well as in the average length of reviews. It seems common sense to expect substantial differences between journals due to differences in fields, journal prestige, and editorial style (although MDPI journals extensively use guest editors). For example, more prestigious journals may attract longer reviews from more knowledgeable or careful reviewers. This finding therefore confirms common sense. Related prior studies have shown differences between journals in reviewer strictness (Hargens, 1990).
F3: Reviewers in the Physical Sciences are most likely to request major revisions. Whilst this does not seem to have been studied before, there are some known disciplinary differences in acceptance rates, with health journals having higher acceptance rates than business, computer science, education, and psychology (Sugimoto et al., 2013).
F4: Reviewers in the Physical Sciences and Social Sciences are most likely to use attachments. This is a relatively minor point. Attachments seem warranted for the Physical Sciences to allow reviewers to include mathematical formulas or symbols. The reason for the higher use of attachments in the Social Sciences is unclear, but perhaps there is a greater need to comment in detail on the exact wording of articles if the social sciences are more subjective.
F5: Physical Sciences reviewers are less likely to sign their reviews. This partly aligns with survey evidence about the reluctance of reviewers to reveal their identities, which found moderately stronger perceived reluctance in Mathematics and Statistics, Physics and Astronomy, Chemistry, and Agriculture and Food Sciences compared to 12 other areas (Ross-Hellauer et al., 2017).
F6: Life Sciences and Social Sciences reviewers tend to write longer reviews. An analysis of 82,798 Publons reviewers, in contrast, found that natural sciences (i.e. life and physical sciences) reviewers tended to write fewer words than those from the social sciences and humanities (Xu et al., 2019). Thus, the life sciences part of this finding may be specific to MDPI.
F7: Reviews with a major revision outcome tend to be longer than those with a minor revision/accept outcome. This does not seem to have been investigated before but is expected because an accept decision has little to report and a minor revision may not need explaining, whereas major revisions probably need to be justified as well as described.
F8: Signed open reviews tend to be 15% longer. This indirectly supports the belief that open peer reviews tend to be higher quality (Rittman, 2014), but partly contracts Bmj (Van Rooyen et al., 1999) and psychology (Jefferson et al., 2002) findings that signed and unsigned reviews are of similar quality, assuming that quality associates with length. Signed open reviews were also more likely to have an attachment, but reviews with attachments were excluded from the length calculations.
Conclusions
The results reveal a range of properties of academic open peer review, some of which align with common sense or previous research, whereas others are more novel and unexpected. About open peer review, the results confirm the strong cross-disciplinary reluctance of reviewers to sign their reviews. Whilst one of the advantages of signed reviews is a potential reduction in bias and another is their apparent extra care (15% longer in this dataset), the apparent consensus against it suggests that it should not be made compulsory except in exceptional circumstances. Nevertheless, the lack of statistically significant association between review outcome and review signing suggests that when used, review signing does not compromise the integrity of the decision.
The validity evidence for the anonymity aspect of open peer review provides support for advocates of the open peer review model but shows that adoption by reviewers is still a major problem that they need to solve, such as through education or incentives. For publishers, the message is that allowing non-anonymous reviews does not seem to compromise review decisions, at least when the reviews are open. This should be encouraging if they are considering switching to an open reviewing model. Giving reviewers the option to remain anonymous seems reasonable at the moment, however, since most reviewers choose not to sign their reviews. For authors choosing where to submit, the possibility of a signed review does not seem to be an important factor to take into consideration since anonymity does not seem to influence outcomes, although the slightly longer reviews of non-anonymous reviewers might be desirable.
The results also show a range of disciplinary differences in reviewing practices for MDPI open reviews. It seems reasonable to at least speculate that these might also apply to other forms of review. This paints a picture of widespread disciplinary reviewing differences, albeit as identified here at a relatively shallow quantitative level. Whilst journals often have guidelines for reviewers, much reviewing knowledge is probably tacit and partly learned through interaction with mentors (e.g. doctoral advisors, supervisors, feedback from previous reviews), and so unexpected differences between journals might occasionally cause reviewers to write inappropriate reviews. The substantial evidence that reviews differ between disciplines in multiple respects suggest that editors and reviewers of articles that cross disciplinary boundaries (e.g. their expertise is in Physical Sciences, but they review a Medical Sciences article) should be particularly careful. This applies to a greater extent to individual journals.
Footnotes
Appendix
The proportion of open reviews with an attachment, by year.
| Year | None | Attachment | Total | Proportion with attachment (%) |
|---|---|---|---|---|
| 2015 | 8 | 1 | 9 | 11.1 |
| 2016 | 83 | 6 | 89 | 6.7 |
| 2017 | 133 | 16 | 149 | 10.7 |
| 2018 | 2640 | 170 | 2810 | 6.0 |
| 2019 | 4493 | 316 | 4809 | 6.6 |
| 2020 | 1872 | 127 | 1999 | 6.4 |
| 2021 | 10 | 1 | 11 | 9.1 |
| Total | 9239 | 637 | 9876 | 6.4 |
Declaration of conflicting interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article is available online.
Author biography
Mike Thelwall, Professor of Data Science, leads the Research Evaluation and Statistical Cybermetrics Research Group at the University of Wolverhampton in the UK.