Abstract
This article reports results of a survey of authors of submissions to Quarterly over a five-year period. The goal was to take stock of who the journal’s contributors are and to get a sense of their evaluation of the peer review process. In addition to describing authors who submit their work (faculty rank, academic degrees, years in teaching, etc.), we chose to examine their views on peer review because of concerns in the literature—in journalism and mass communication (J/MC) and the academy in general—with the process.
Studies have shown the process can have inconsistent standards, 1 an inability to catch mistakes 2 or detect fraud, 3 a confirmatory bias, 4 and even bias against female authors. 5 Given demographic shifts in the academy—women outnumber men among those earning doctorates, though they remain outnumbered among tenure-track faculty 6 —examining perceived fairness of peer review is timely. This study investigates perceived biases against specific research approaches, but also perceived gender bias. 7
We also see these data as providing an “update” to a number of previous studies of J/MC scholars’ perceptions of peer review. Ryan compared journal referees and J/MC faculties’ rankings of sixty evaluation criteria. 8 While Ryan asked about peer review “in principle,” Leslie had AEJMC (Association for Education in Journalism and Mass Communication) members report their levels of satisfaction with peer review “in practice.” 9 Leslie called it “startling—and troubling” that the highest-rated practice is essentially clerical: “acknowledgment of receipt of your article”! Leslie also solicited open-ended responses, which included descriptions of an “old boys” network and comments that J/MC research is “too quantitative” and “based on social science methodologies.” Surveying AEJMC members, Poindexter found women more likely than men to rate “bias against methods” and “bias against topics” as threats to the peer review process. 10
Surveys in other disciplines have found the strongest predictor of satisfaction is simply whether one’s latest manuscript was accepted. 11 The authors concluded that peer review is seen as a “hurdle” rather than an opportunity to obtain advice and assistance. 12 This “hurdle vs. opportunity” issue has not been addressed within J/MC. Thus, we pose three research questions:
Method
Design and Sample
Authors who had submitted at least one manuscript to JMCQ from 2005 to 2010 were invited to complete a web-based survey about the review process for the last article they submitted to JMCQ and about review processes of other mass communication journals. A link was emailed in fall 2010 to 714 authors; five reminders yielded 377 (52.8%) responses (330 were fully completed). To secure human subjects approval, all identifying information was removed once survey responses were received. No identifying information was ever shared with any journal personnel, a fact emphasized to respondents.
Respondents indicated the decision (accept, reject, revise) on their last JMCQ submission, how many articles they submitted and had accepted by JMCQ and other mass communication journals during 2005-2010, and their total career peer-reviewed publications (as sole or co-authors). They reported hours per week spent on research and percentage of work effort devoted to research. Each identified a preferred research approach: qualitative, quantitative, mixed-methods, or “other.” Finally, respondents indicated their age, sex, years of experience in higher education, highest degree, academic rank, and tenure status.
To measure beliefs about the peer review process, respondents used a 7-point scale (7 = strongly agree) with seventeen statements adapted from previous studies 13 about peer review (see Table 2 for wording and descriptive statistics), ranging from administrative processes to the substance of reviewer comments, to perceptions of bias and whether reviewer comments were helpful in improving one’s work. Respondents completed the battery of seventeen items for “mass communication journals” before completing the same battery for JMCQ.
Based on principal components analyses, eleven of the seventeen items loaded on a single “Overall Satisfaction” component for both JMCQ (α = .972) and “mass communication journals” (α = .952), while three loaded on an “Overall Bias” component (JMCQ, α = .917; “mass communication journals,” α = .837). While three questions failed to load, one (“I was satisfied with the time it took to complete the initial review of my submission”) was retained as a single-item measure.
Results
Descriptive Statistics on Academic Variables, Age and Years on Faculty, and Research Activity, by Gender.
Note. JMCQ = Journalism & Mass Communication Quarterly.
Means joined by common underscore are significantly different, by t-test, p ≤ .05.
Respondents averaged 15.5 articles published in peer-reviewed journals across their career, had submitted 9.9 articles to peer-reviewed journals in the past five years, and had 7.5 accepted. Each respondent’s five-year “success rate” was computed as follows: number of manuscripts published divided by number submitted—the sample-wide average was 0.711 with a SD = 0.272.
They had submitted an average of 1.8 articles to JMCQ in the past five years, of which 0.70 had been accepted (average JMCQ “success rate” = 0.325, SD = 0.411). The fact that half are at senior rank and tenured, and that all had submitted a manuscript to JMCQ in the past five years, suggests that these are experienced scholars.
Table 1 data also reveal similarities, and some differences, between male and female contributors. Notably, there are no statistically significant differences on a number of individual characteristics—tenure status, rank (despite a greater percentage of female graduate students and assistant professors), degree attainment—as well as time (in hours per week) and effort (as a percentage of total) devoted to research and preferred research approach. Generally speaking, male and female contributors to JMCQ have been similarly active as researchers during the last five years (in terms of similar numbers of articles submitted and published in JMCQ and in other peer-reviewed journals).
Career-wise, however, men in the sample had significantly more total refereed journal publications (17.5) than women (12.9), a “gender gap” in productivity that mirrors age differences: female respondents were significantly younger than their male counterparts and had significantly less academic experience. Judging from self-reported data for this sample, then, any gender difference in research productivity is a function of the age and greater accumulated experience of the males in the sample.
Note. JMCQ = Journalism & Mass Communication Quarterly.
Items rated on a 7-point Likert-type scale (1 = strongly disagree and 7 = strongly agree).
Asterisks indicate significant differences; differences between means were tested with paired-samples t-tests; differences between percentages were tested using chi-square tests.
p < .05. **p < .01.
JMCQ was viewed more positively than mass communication journals in terms of administrative performance—the time it takes to complete the initial review of a manuscript (JMCQ, M = 4.44; mass communication journals, M = 3.86) and correspondence explaining the editorial decision (JMCQ, M = 4.63; mass communication journals, M = 4.41). Authors also rated JMCQ more positively for clarity (JMCQ, M = 4.75; mass communication journals, M = 4.56) and politeness of reviewers’ comments (JMCQ, M = 4.96; mass communication journals, M = 4.72).
Authors rated JMCQ more negatively in terms of how helpful peer review was for the reviewed manuscripts (JMCQ, M = 4.31; mass communication journals, M = 4.56), subsequent manuscripts (JMCQ, M = 3.98; mass communication journals, M = 4.34), or their scholarship as a whole (JMCQ, M = 4.32; mass communication journals, M = 4.62).
Comparing item means, however, tells only part of the story. Examining the percentage data contrasting JMCQ with “mass communication journals,” one discovers that authors were more likely to “strongly agree” with eight out of thirteen individual statements about JMCQ’s peer review process. However, there was also a significantly larger percentage of respondents who strongly disagreed with the statements that “JMCQ’s peer review process has enhanced my scholarship” (JMCQ, 18.8%; mass communication journals, 10.6%) and “JMCQ reviewers’ comments have been helpful in improving my subsequent work” (JMCQ, 23.0%; mass communication journals, 12.7%) than the percentage who agreed.
Correlations (Pearson’s r) between Key Peer Review Variables and Author Characteristics (Pairwise Deletion Used; No of cases = 267-325).
Note. JMCQ = Journalism & Mass Communication Quarterly.
Dummy-coded; 1 = rejected, 0 = accepted or invited to revise and resubmit.
Dummy-coded; 1 = tenured, 2 = not tenured.
Dummy-coded; 1 = female, 0 = male.
Dummy-coded; 1 = qualitative or mixed-methods researcher; 0 = quantitative researcher.
p < .05 level. **p < .01 level.
Dependent measures in the models were the summed “Overall Satisfaction” (Model 3) and “Overall Bias” (Model 2) measures, and satisfaction with the time of the review (Model 1). The first two models use as predictor individual characteristics and whether the last manuscript was rejected (see Table 4). The third model used the individual characteristics, whether the submission was rejected, and assessment of the time for review and the overall bias score. Whether quantitative research was one’s preferred approach was also used as a predictor because previous surveys raised the concern, particularly among female respondents, that J/MC journals are biased against non-quantitative methods.
Regression Models for Author Satisfaction with JMCQ’s Peer Review Process.
Note. Standardized betas. JMCQ = Journalism & Mass Communication Quarterly.
Dummy-coded; 1 = female, 0 = male.
Dummy-coded; 1 = qualitative or mixed-methods researcher; 0 = quantitative researcher.
Dummy-coded; 1 = tenured, 2 = not tenured.
Dummy-coded; 1 = rejected, 0 = accepted or invited to revise and resubmit.
p < .05. **p < .01 level. ***p < .001 level.
As shown in Table 4, the first regression model predicted 10.3% of variance in authors’ satisfaction with the time required to review their submission to JMCQ. Being female (β = .158, p < .01) and tenured (β = .118, p < .05) predicted greater satisfaction with the time for the review. Being tenured in particular may make an individual less anxious/more patient to receive a peer review decision.
Having the submission rejected predicted an additional 6.4% of the variance in authors’ satisfaction with the review time. Authors whose manuscripts were rejected after the initial review were significantly less satisfied with the time it took to receive that decision (β = −.253, p < .001), perhaps reflecting a preference for receiving even a negative decision quickly to move on and resubmit the manuscript elsewhere.
The second regression model predicted 14.2% of variance in authors’ perceptions that JMCQ’s peer review process is free of “Overall Bias.” Individual characteristics predicted 5.5% of variance. Non-quantitative researchers were significantly less likely to perceive the review process as unbiased (β = −.209, p < .001), a relationship that merits expansion. Non-quantitative authors (60%) in the sample were in fact more likely to report having their last article rejected by JMCQ than were quantitative authors (40%), but that difference was not statistically significant, χ2(1, N = 317) = 3.174, p > .05.
However, if one uses t-tests to examine differences in computed “success rates” over the past five years, quantitative researchers’ success rates for JMCQ are significantly higher (M = 0.424, SD = 0.423) than they are for non-quantitative researchers (M = 0.245, SD = 0.383), t(1,313) = 3.954, p < .001. The opposite is true for mass communication journals generally, quantitative M = 0.687, SD = 0.269; non-quantitative M = 0.746, SD = 0.258; t(1,313) = −1.979, p < .05.
The outcome of the peer review process explained an additional 8.7% of the variance in authors’ perception that JMCQ’s peer review process is unbiased. Authors whose manuscripts were rejected were significantly more likely to view the process as biased (β = −.295, p < .001).
Finally, the third regression model predicted 58.7% of the variance in authors’ “Overall Satisfaction” with JMCQ’s peer review process. Individual characteristics explained 3.5% of the variance. However, only author gender was significant: female authors are more likely to be satisfied with the journal’s peer review process (β = .101, p < .05).
Author satisfaction with the other dimensions of peer review—the length of the review process and the lack of bias—was important, explaining an additional 45.7% of variance in “Overall Satisfaction.” Satisfaction with time of review (β = .286, p < .001) and perceived lack of bias (β = .404, p < .001) were both very positively associated with “Overall Satisfaction.”
Finally, whether an author’s manuscript was rejected explained an additional 9.6% of variance; rejected authors were significantly less satisfied with the process (β = −.329, p < .001).
We began this report by considering whether author gender would relate to perceptions of the peer review process. As shown above, female authors had more positive perceptions of JMCQ’s peer review process: in Model 1, they were significantly more likely to be satisfied with the time it took to review their manuscript (β = .158, p < .01), and they were in Model 3 more satisfied overall (β = .101, p < .05). However, gender was not significantly associated with Model 2 perceptions of bias in JMCQ’s peer review process (β = .011, p > .05).
Given the significant effects of both gender and editorial decision, it makes sense to probe further the relationship between these two variables. Table 5 examines gender, acceptance or rejection, overall satisfaction with the peer review process, and perceptions that the peer review process improved one’s work. A one-way analysis of variance (ANOVA) shows a significant relationship among gender, manuscript acceptance/rejection, and overall satisfaction with the peer review process. Post hoc tests show no significant difference in overall satisfaction between male (M = 5.24) and female authors (M = 5.50) whose articles were accepted.
Gender Differences in Satisfaction with the Peer Review Process by Whether Submission to JMCQ was Rejected (one-way ANOVAs).
Note. All variables were scored on a 7-point Likert-type scale (1 = strongly disagree, 7 = strongly agree). Because of unequal variances, Games-Howell post hoc tests were used to test for significant mean differences between groups; common subscripts indicate significant between-group mean differences in rows. JMCQ = Journalism & Mass Communication Quarterly; ANOVA = analysis of variance.
Omnibus F significant at p ≤ .001.
However, men who had articles rejected had a more negative reaction and were significantly more negative in rating the peer review experience (M = 3.45) than their female counterparts who were rejected (M = 4.14). Indeed, female authors who had articles rejected were more likely than males who had articles rejected to say that the JMCQ peer review process nonetheless improved their current manuscript (female, M = 3.93; male, M = 3.14) and their subsequent work (female, M = 3.78; male, M = 3.08).
Conclusion
This study found that authors who have submitted to JMCQ in the past five years had slightly more positive than negative attitudes toward the journal’s peer review process, but that the review process was rated less positively for helping improve authors’ current manuscripts or subsequent work. Data suggest that JMCQ’s peer review process is not seen as particularly better or worse than the processes of other mass communication journals, except in two regards: as noted, JMCQ reviews are rated as somewhat less helpful, but the journal’s reviews are perceived as clearer, more polite and professional, and certainly faster.
The regression analyses also show that whether an author’s previous submission to JMCQ was rejected was consistently one of the strongest predictors of satisfaction with the peer review process. Again, this finding raises troubling questions about the perceptions of peer review as a hurdle rather than a means to improve one’s scholarship. 14
Rejection also related to perception of bias. Non-quantitative researchers were less likely to perceive JMCQ’s peer review process as unbiased, and non-quantitative researchers had significantly lower rates of success publishing in JMCQ, but significantly higher rates of publishing success in other mass communication journals.
Women, however, actually rated the peer review process more positively, even when the editorial decision was negative. Women who had their articles rejected were significantly more satisfied with the peer review process and more likely to credit it with a positive impact on current and subsequent research than were men who had their last article rejected.
These gender differences are consistent with previous research on women’s responses to evaluation that suggests that women internalize negative feedback (i.e., view a negative review as reflecting weaknesses in one’s manuscript) rather than blaming others (i.e., reviewers) or external factors or processes, 15 are more eager to use negative evaluative feedback to improve future performance, 16 and, if they receive negative feedback, actually are more likely to improve subsequent performance than are men who receive negative feedback. 17
Of course, the results of this study cannot be generalized beyond the survey’s respondents. The survey focused on authors who had submitted to a single journal, and the authors were not randomly selected. Nonetheless, the data may be instructive for those who review for mass communication journals and those who make reviewer assignments. The data may also provide some context for authors submitting their work for peer review. Furthermore, while this study was not set up as a theoretical test of gender differences in response to evaluative feedback, it does suggest some avenues for future research of potentially significant gender differences beyond concerns of bias.
Footnotes
Authors’ Note
This editorial report was not peer-reviewed. However, a longer version of the survey results was reviewed and presented at a session sponsored by the Commission on the Status of Women at the AEJMC conference in Chicago in August 2012.
Declaration of Conflicting Interest
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.