Abstract
This essay provides advice for effectively reviewing conference proposals, including how to write comments that are helpful to proposal authors, how to use the “Comments to Program Chair” box, and issues to consider when assigning proposal ratings and recommending acceptance or rejection. Several benefits of reviewing proposals are outlined along with advice for becoming a reviewer. This essay is situated within the American Educational Research Association conference context and considers how reviewing conference proposals differs from that of journal articles. Still, much of the advice provided is applicable to scholarly reviewing, more generally.
The number of proposals submitted for the American Educational Research Association (AERA) annual meeting has grown dramatically in past decades. In 1976, AERA reported a record number of submissions with nearly 2,000 proposals submitted (AERA, 1976). By 1982, the number reached over 3,000 (AERA, 1982) and then neared 5,000 at the turn of the century (AERA, 2000). With advances in electronic submissions, by 2008 there were 12,000 submissions (AERA, 2008). Since then, proposal numbers have increased more slowly, averaging roughly 13,000 the past 3 years. 1 Overall, since 1976, the number of proposal submissions has increased more than sixfold, far outpacing the growth in AERA’s membership. 2 Hence, it is not surprising that securing enough proposal reviews is sometimes a challenge for AERA conference organizers, as reviewers are in demand now more than ever.
AERA requires proposals to be submitted to one of three entities: (a) a specific AERA division (or section within a division), (b) an AERA committee, or (c) a special interest group (SIG). Regardless of where a proposal is submitted, program chairs are expected to obtain at least three reviews for each proposal.
As part of my duties as program chair of a relatively large SIG, I had the privilege of managing its review process for the 2019 annual meeting. Thanks to the generous efforts of 124 reviewers, I obtained roughly 800 reviews of the 163 proposals submitted. The review timeline is fairly tight, allowing only a few weeks for reviewers to complete their reviews and then 6 weeks for program chairs to use those reviews to make decisions about which proposals to include in the program and whether accepted proposals will be part of a paper, roundtable, or poster session. Clear, insightful proposal reviews are essential for good decision making and for helping authors in our field push their work forward.
Similar to most conference review processes, AERA provides reviewers with a set of criteria for rating each proposal, including its objectives, methods, and significance. There is also a short video available to reviewers, which briefly outlines these aspects, as well as more general issues, such as handling conflicts of interest. 3 Despite this guidance for reviewers, I noticed that the length, tone, and scope of the reviews I received varied widely. I found some reviews especially helpful, and I began to analyze the characteristics of those strong reviews as well as the weaknesses in other, less helpful reviews.
Writing good reviews is an important skill that can be improved with practice and guidance. Several journal editors have provided advice for writing good article reviews. For example, Silver (2003) notes the need for reviews to play a dual role of being educative as well as evaluative. With a focus on the educative role, Crespo (2016) draws from Elbow’s (2000) framework of doubting versus believing, as she argues that reviewers should avoid approaching a manuscript in search of errors and omissions (doubting) and instead assume the author’s claims have merit (believing) and look for ways to strengthen and improve them.
Although some of my own observations about good reviews echoed Crespo’s (2016) and Silver’s (2000) comments about reviewing for journals, I was also struck by aspects that are specific to the conference reviewing context. For example, with conference proposals, authors typically have only a few pages in which to communicate their ideas, and there is no revise-and-resubmit option—the program chair must make a binary accept/reject decision but must also decide the mode of delivery (e.g., paper or poster) for accepted proposals.
Given that conference proposal reviewing can serve as a first foray into scholarly reviewing (e.g., graduate students composed one third of my reviewer pool), it seems especially useful to provide guidance about reviewing in this arena. My hope is that those who seek to become a better reviewer, as well as those mentoring junior scholars, will benefit from the discussion here. I begin with a focus on writing reviews that are helpful to conference proposal authors and to the program chair and then briefly address review length, numerical ratings, and the benefits of reviewing.
What Makes a Review Helpful to the Author?
AERA and many other conferences ask reviewers to submit their main, written evaluation of a proposal in a “Comments to the Author” box. These comments can be extremely heartening or disheartening to proposal writers. However, even reviewers who recommend rejecting a proposal can provide helpful encouragement to the author. The reviewers who are most helpful to authors do the following.
Focus on the Most Important Issues
Instead of listing dozens of small concerns, good conference proposal reviewers focus on the most important, substantive issues when writing comments to the author. For example, although reviewers might mention overall concerns about writing clarity, there is no need to focus on typos or other minor errors. Again, the goal is not to help the proposal become polished prose for publication but to provide helpful feedback to the author about their research and to inform the program chair’s decision.
Be Constructive
Aligned with observations from Crespo (2016) and Silver (2003), good reviews are educative, giving the author clear directions for improvement and next steps. One thing that stood out to me when reading the hundreds of reviews was that the most constructive reviews had an air of optimism, even when containing serious critique, using phrases such as “This study holds promise for . . . ,” “This study does not yet provide sufficient depth . . . ,” or “The study could do even more to highlight . . . .” 4 Helpful reviews provided clear ways to address the most serious weaknesses in the work and move it in a more fruitful direction, such as recommending key citations to relevant studies or specific analyses of their data that could enrich the work. Some reviewers used “I statements” to soften their criticism (e.g., “I am not sure why this site was chosen” or “I would have liked to see . . .”).
Good reviewers balance critique with positive comments. For example, even when faced with a seriously flawed proposal, reviewers might acknowledge the good intentions and efforts of the authors (e.g., “This is an ambitious study of an important topic” or “The range of data collected is impressive”). Given the strict word limits for most conference proposals, reviewers might also acknowledge that space constraints may have contributed to weaknesses or missing information in the proposal.
Sarcasm and sweeping, negative statements (e.g., “There is no research here”) have no place in scholarly reviews. Those who tend to get rather passionate while reviewing may need to pause for a while and reread their reviews before submitting. Reviewers might also ask themselves, “Do I recognize at least some merit in the work?” “Do I phrase criticism constructively, providing clear guidance for addressing weaknesses?” “Are my comments likely to demoralize the researcher instead of equipping them to improve their work?
Look Beyond Personal Passions and Pet Peeves
Conference proposal reviewers are often assigned a broad range of proposals, and good reviewers recognize that authors may have valid perspectives and goals that might not match their own. Reviewers should not let their own research agenda or pet peeves serve as their only lens when reviewing. A good reviewer might have a passion for structural equation modeling or poststructuralist frames, but they carefully consider the merits of what is there before dismissing a proposal for lacking their preferred approach or framework.
Look Beyond the Conference Theme
In a similar vein, reviewers should not automatically dismiss proposals for lack of fit with the conference theme. A reviewer may mention particularly strong alignment with the conference theme as a strength of a proposal, but most large conferences, such as the AERA annual meeting, will have a breadth of topics. Although the theme can serve to highlight research in a particular area and provide a unifying thread across selected sessions, the theme need not be viewed as restrictive, unless that is clearly the conference organizers’ intent.
Proofread
A review that is clear and carefully constructed not only is helpful to the author but also enhances the author’s confidence that the review process was carefully executed by knowledgeable reviewers. Hence, reviewers should read through their reviews with an eye toward clarity and accuracy. This is especially important when issues of clarity and grammar are part of the reviewer’s critique of a proposal.
How Should the “Comments to the Program Chair” Space Be Used?
In addition to the “Comments to the Author” section on AERA’s proposal review form, there is also a space for “Comments to the Program Chair.” This is common in many reviewing contexts, whether for conferences or journals. As program chair, I noticed that many reviewers wrote “See author comments” in the program chair box or copied their comments to the author and pasted them into the program chair box. Several reviewers even painstakingly reworded all of their comments to the author before moving them to the program chair box, changing the text from the second person (“You need to . . .”) to third person (“The author needs to . . .”). None of these approaches is actually helpful. Instead, reviewers should do the following when writing comments for the program chair.
Be Brief
Lengthy comments to the chair that repeat comments made to the author only adds to the chair’s reading load. As program chair, I appreciated when a reviewer wrote one or two sentences that summarized the study’s focus and contribution and briefly explained why they recommended acceptance or rejection of the proposal.
Be Blunt
Although comments to the author should be worded gently, in the interest of brevity, the comments to the program chair can be quite frank. These comments are for the chair’s eyes only, and so reviewers should feel free to give a blunt assessment of the proposal’s quality, contribution, and potential interest to the community. This is a space to say, “The authors found several correlations, but the results tell us what we already know,” or “I read the proposal twice and still am not sure what this study is about.” Reviewers can also give a blunt assessment of their own expertise as it relates to the proposal. For example, a reviewer might note, “I am not an expert in multilevel modeling, so I defer to other reviewers for assessing whether the methods were used appropriately.”
Be Consistent With Comments to the Author
Although comments to the program chair can be very blunt, they should be consistent with the content of the overall review. For example, a reviewer should not tell the author, “This is an interesting proposal,” but then tell the chair, “This work has already been done.”
Address the Conference Context
I was surprised to see reviewer comments such as “If the author addresses my feedback, then the paper should be accepted.” Such comments point to a major difference between the role of a program chair and journal editor—that is, a program chair has no editing role and only one decision point, as there is no revise-and-resubmit process. However, a program chair does decide the form of an accepted presentation (e.g., paper, poster, or roundtable presentation). Reviewers might use the “Comments to the Program Chair” space to make a specific suggestion about the form a presentation should take. For example, a reviewer might note, “The findings challenge conventional wisdom and are likely to generate important discussion—put in a paper session if possible.”
Additional Issues
Reviewers can also use the “Comments to the Program Chair” space to alert the chair to any other pertinent issues. For example, I appreciated one reviewer’s frank note indicating that two very similar proposals from the same project were submitted and recommending that only one be accepted. However, issues that are urgent because they might require immediate investigation or reassignment of a proposal are best handled via an email to the program chair. For example, I received several queries from conscientious reviewers as they began their reviews, including questions related to an author’s blatant disregard for word limits and blinding protocol as well as potential conflicts of interest.
How Long Should Reviews Be?
Whereas reviews of journal manuscript submissions are often quite lengthy (e.g., one to three pages), there are two reasons why reviews of conference proposals are typically more brief. First, conference proposal reviewers are often asked to review five to 10 or more proposals within a short time frame, which limits the time reviewers can spend on each review. Second, for many conferences (including AERA), the relatively brief proposal itself does not need to be perfected for publication. Hence, reviews need not help the author refine the proposal’s text but instead should provide general feedback about the direction of the work and help the program chair decide whether the work merits presentation. Hence, a couple of thoughtful paragraphs to the author and a few sentences to the program chair can be sufficient.
What About Ratings?
Most conference organizers ask reviewers to use a rubric and rate specific areas. For example, AERA asks reviewers to rate six categories on a scale of 1 (low) to 5 (high):
Objectives or purposes
Perspective(s) or theoretical framework
Methods, techniques, or modes of inquiry
Data sources, evidence, objects, or materials
Results and/or substantiated conclusions or warrants for arguments/point of view
Scientific or scholarly significance of the study or work 5
Additionally, AERA reviewers must make an overall recommendation for whether proposals should be accepted or rejected. On the basis of my observations of the varied ways in which reviewers rated proposals, I offer the following advice.
Vary Your Ratings
Some reviewers seemed to assign a high rating to each category by default (e.g., 4 or 5 on the 5-point scale) with a recommendation of acceptance unless there were very grave concerns. Although this may seem encouraging to authors, it does not really help the author or the program chair understand their proposal’s strengths and weaknesses. Additionally, overuse of high ratings makes it difficult for program chairs to efficiently rule out weaker proposals and focus on those that merit further consideration. Although a few excellent proposals may deserve top ratings in every category, in general, reviewers should strive to vary their ratings both within and across proposals to indicate where a proposal is strongest, what aspects could be improved, and which proposals should (and should not) be included in the conference program.
Make Your Overall Recommendation Consistent With Your Ratings and Comments
Reviewers should ensure that their comments to the author and their ratings for the six categories align with their recommendation regarding acceptance. A reviewer who assigns mostly 1s and 2s to a proposal should recommend rejection, whereas a proposal with mostly 4s and 5s should be recommended for acceptance. If a proposal has a mix of ratings, then thoughtful reflection on the proposal’s overall significance (AERA’s sixth review category) can help the reviewer solidify their recommendation. When determining significance, reviewers might ask themselves, “Is this research important for presentation at this conference?” “Does it rigorously build upon and extend prior research in the field?” “Does it provide new, helpful insights into key problems of research or practice?”
Why Be a Reviewer?
Reviewing can seem like a thankless job, but there are benefits of reviewing, whether it be for a conference, journal, or grants program. First, reading others’ proposals can help you recognize what makes a strong or weak proposal, thereby helping you improve your own writing skills (Silver, 2003). Second, given that proposals are usually matched with reviewer expertise, those who review gain the opportunity to see new research in their area of interest well before it is published. Finally, although reviewing is largely behind the scenes, reviewing can actually help “get your name out there.” Certainly, the program chair gets to know your name and expertise while assigning and reading reviews. Additionally, many organizations and journals publicly thank their reviewers and even give awards for outstanding reviewers.
How Do I Become a Reviewer?
Most conference program chairs are happy to have both graduate students and more established scholars in their reviewer pools. Reviewer criteria may be more stringent for journals and funding agencies, but qualified reviewers can usually volunteer to review for journals via their websites, and for funding agencies through emailing a relevant program officer.
Within AERA, a call for conference proposal reviewers goes out to AERA members each spring. Volunteers who want to maximize their chances of being called upon to review may volunteer for more than one division or SIG. However, one caution is warranted: Program chairs can see the total number of proposals you have volunteered to review, and they may be reluctant to assign proposals to you if they fear you are overcommitted. 6 Hence, reviewers who want to gain reviewing experience should sign up to review for only the most relevant AERA units but then broadly indicate all areas of their expertise, as there is more demand for experts in some topics than others.
Final Thoughts
In closing, the field benefits greatly from good reviews, and reviewers can reap benefits from assessing others’ work. I learned a great deal from the many thoughtful reviewers who informed my decisions as program chair. I hope this discussion of conference proposal reviewing will entice others to review and will help reviewers work efficiently, maximizing the benefits of their reviews while limiting time spent on nonessential aspects.