Reviewer #2 Must Be Stopped!
The Art of Providing Good Reviews.
Steve Simske, Alexandra Bonnici
As of 2022, over 5.4 million academic articles are published each year. These include short surveys, reviews, conference proceedings and journals. The gatekeeper that ensures the quality of these articles are peer-reviews: a tried-and-tested way through which other expert academics decide on the relevance and contribution of the article to the general knowledge in the subject area. And while we, as authors, may not particularly be overjoyed at receiving a negative or critical review, there is a difference between a review that offers a critical and meaningful review and one that is just a bad review: the reviews that are one-liners, offering no help, to the downright snarky ones that leave authors disheartened with the paper publishing process.
If there are reviewers who provide good reviews and others who provide less than stellar ones, then how do conference chairs go about selecting a suitable group of reviewers?
PhD comics neatly sum up how reviewers are selected. Only, this is not the bad thing that it is made out to be in the comic strip. A program committee is made up from people from within the community and recruiting new reviewers from the previous conferences is one way to keep rejuvenating the programme committee. But there is some truth in the comic strip. How does one provide a good review if one does not have, or consider themselves to have sufficient experiences? Can we train young researchers in conducting effective peer-reviews?
Note that in this report, we are not advocating for non-critical reviews. If a paper has flaws, it is important that these are highlighted so that authors may address them. What we would like to advocate for is for reviews to be a teaching experience.
Example 1
Even the authors could not clearly explain in the conclusion or in the body what are new results. The article is not suitable even for a poster, since it is in the very early stage of the investigation. Numerical experiments are not of sufficient interest for a research article.
Example 2
The authors apply some standard approach to some unavailable data set and produce some numbers. Where is novelty? This is just a report on some work and is not a suitable paper for this conference!
Example 3
Great paper; recommend acceptance.
Reviews such as those of Examples 1 and 2, while potentially correct within the context of the paper, do not really explain to the authors what could have been done better. The reviews belittle the authors' work, and both may be extremely disheartening for the authors receiving the review. The third example, while positive, is also a bad review! It offers no insight on what makes the paper such a great one, making it difficult for the program chair to understand the justification behind this recommendation. It is these types of reviews that we feel must be stopped.
The Peer Review
The peer review is a quality control mechanism that subjects an article to the scrutiny of independent and impartial experts in the respective fields. These experts, referred to as the peers or reviewers, will judge the article for its content, methodology, originality, and the overall significance of the work. The reviewers will also provide feedback and recommendations to the authors to help the authors improve their work.
The peer review process is therefore a quality assurance process, subjecting the authors’ work to expert evaluation that exposes errors, inaccuracies or deficient that must be resolved before the article is presented for general dissemination. This, in turn, ensures that the research published is both reliable and credible.
The scrutiny of the reviewers ensure that the methodology adopted are theoretically appropriate and technologically sound, hence validating the findings reported in the paper. Through the peer review, we have the assurance that results are robust and interpreted appropriately.
The constructive criticism provide by the reviewers provides the authors with feedback that helps the authors to clarify arguments or improve methodology with the scope of providing a more impactful and meaningful contribution to knowledge.
In short, the peer review engages the community to build upon each other’s knowledge and experiences to collectively advance knowledge in a specific area. This requires a community of reviewers that are objective, impartial and genuine in their intent to develop the research area. Providing this kind of feedback takes time and effort, but done properly, the peer-review process offers several benefits to authors, the reviewers and the research community.
Benefit to Authors
For authors, the peer review process gives the assurance of improved quality of the contribution and research. Even if the paper is rejected, good reviews will provide the authors with invaluable insights on how the paper, and their research, may be improved for the next submission of the paper. This helps the authors submit stronger contributions.
The peer review process will also result in publications that can be trusted. Knowing that a paper would have passed through the scrutiny of experts assures researchers reading the paper that the work meets certain standards. This lends credibility and validation to the authors’ work. This credibility implies that peer-reviewed works are more likely to be read and subsequently cited, thereby increasing the visibility and recognition of the authors within the academic community. The exposure and recognition garnered would indirectly increase the opportunities for collaborations, exposing the authors’ work to other researchers in the academic and professional community.
Benefit to Reviewers
For reviewers, the peer-review process means that the reviewers have a direct contribution to the field, giving the individual the opportunity to have an active role in shaping the quality and direction of research in the field. Through constructive feedback the reviewers are helping authors improve their work and hence, contribute to the advancement of knowledge. In providing good quality reviews, the reviewer must keep abreast with the current state-of-the-art and good practices in the field. Through providing the review, the reviewers, particularly young reviewers, are enhancing their critical thinking and analytical skills to evaluate the strengths, weaknesses and contributions of the works being evaluated. This results in personal growth of the reviewer as an academic professional.
Although the PhD Comics strip made fun of asking the previous year authors to act as reviewers, this is generally a sign that the individual’s academic contributions are being recognised. Serving as a peer reviewer is then a sign that the expertise and reputation of the individual are being appreciated by the academic community.
Benefit to the Academic Community
For the academic community, and in particular, program chairs or journal editors, the peer-review process ensures that decisions are reached based on consensus among an admittingly small sample of the research community. The peer-review then establishes a small communication and discussion link between field experts to address controversies and reach consensus, assisting the editor or program chairs in determining whether the submitted work should be accepted or otherwise.
The Review Process
Using the ACM Symposium on Document Engineering as a Case Study
Having gone over the benefits of the peer review, we can now take a closer look at the peer-review process, using the ACM Symposium on Document Engineering (DocEng) as a case-study.
The process would start with the selection of the program chair and the program committee. These are typically selected way before the call for papers are issued. Individuals who served on the program committee in the previous year are invited to participate in the program committee, adjusting the committee members by removing the members who are not available due to other commitments, or to remove reviewers whose performance was not up to par.
At DocEng, authors are asked to first submit an abstract of the paper, around a week before the manuscript submission date. The abstracts are used by the reviewers to bid on the papers that they would like to review, ranking each paper as being within their interest, not fully within their expertise, but would still be ready to review, and not ready to review. This decision is not made based on favouritism, but based on the reviewer’s expertise in the topic area. This ensures that each paper can be reviewed by individuals who are experts in the area and can therefore provide a good review to the paper.
In the interim period between the paper bidding and the author submission of the full paper manuscripts, the program chair reviews the bids and pairs three reviewers with each paper, according to the reviewer preferences. Once all the papers are submitted, the reviewers are give around a month to carry out the reviews independently of each other. During this phase, the reviewer will not be able to see the reviews submitted by the other reviewers, so as to ensure the complete independence of the review process.
Once the review has been submitted, the program chair reads through it to verify that the language used is appropriate and, if necessary, asks the reviewer to revise the review to use more appropriate language. Once a review on a paper has been submitted, the reviewer is given access to the reviews of other reviewers. This is a necessary step since it allows the reviewer to gauge their own review against the comments of the other reviewers. In fact, once the three reviews have been submitted, the program chair invites the reviewers to make any further comments, particularly if the reviews have conflicting recommendations. These additional comments are not made visible to the authors, although a reviewer may opt to change their initial review to reflect the consensus reached from these additional comments.
The program chair will then see the reviews and the additional comments, considering all the reviews and the relevance of the papers to the conference and the overall conference program to make a decide on whether to accept or reject a paper.
DocEng has two paper tracks: the full papers and short papers, with promising, yet rejected full papers being invited to resubmit as short papers. Thus the review process is repeated for the short papers, albeit with an additional consideration. For resubmitted papers, we attempt to retain part of the original reviewing group, while injecting new reviewers for a fresh pair of eyes. In this manner, there would be visibility over the original comments, while allowing for a fresh review to take place.
The Single-Blind Review
The review process at DocEng follows the single-blind review process. Here, the author names are made accessible to the reviewer, but the reviewer names are not disclosed to the authors, although during the review process, the reviewer names are disclosed to the other reviewers.
There are several advantages offered by the single-blind review:
The reviewer does not have to fear from any form of retaliation from the authors and can, therefore, be critical and open in their review. This freedom ensures that the reviews provided can be candid and constructive, reducing the bias that may arise in situations when authors are from established universities or research groups, or the authors themselves are well established in the field. By not disclosing the author names, younger researchers, or from lesser established universities or research groups can still make comments based on the quality of the paper at hand. This is not only necessary for the merits of the papers by any such authors, but also ensures that all papers can be treated with the same level of critical analysis. Anonymity of the reviewer places all reviewers at the same and level playing field.
In the single-blind review, the author names are known to the reviewer. This too has some advantages. It facilitates the reviewer's understanding of the context and background of the author's previous research which would help with the assessment of new contributions and knowledge advancement over the author's own previous works. Knowing the author names would prevent assumptions about the previous works or any biases in the evaluation of the paper.
From the authors' side, having their names made known to the reviewers encourages a degree of accountability from the authors who may be more inclined to be accurate and rigorous in their works, potentially leading to better quality submissions knowing that their names and subsequently reputations are known to the reviewers.
With the reviewers knowing the author names it will also be easier for reviewers to flag any potential conflicts of interest with author groups with whom there was prior collaboration or history. This too ensures transparency and lack of favouritism among the authors and reviewers.
These same advantages may, however, pose some disadvantages. Mainly, safe in the knowledge that their names are not disclosed, reviewers may be tempted to be somewhat rude or overly critical in their review. Moreover, there is a risk that, knowing the author names may lead to unconscious bias towards the reputation of the authors, favouring, or being unconsciously less critical to works from authors of well established groups and more critical to others from lesser-known institutions. Other biases based on race, gender or geographical location may also unconsciously set in, undermining the quality of the reviews. By being aware of these disadvantages, it becomes possible to have some checks in place to minimise the possibility of unfair biases that may occur.
At DocEng, care is taken to recruit a largish pool of reviewers so that reviewers are not overloaded with papers to review. Typically, a reviewer would have a load of two to three full papers and two to three short papers. Combined with this, reviewers are allocated around a full month to complete the reviews. Although the reviewing load does not require a month's work, this does allow reviewers to plan ahead and schedule quiet time to carry out the reviews. Reviewers are informed at the time of invitation of the tasks expected from them so that commitment is made with full knowledge of expectations. Besides distributing the workload, the largish pool of reviewers also provides diversity in the expertise and experiences of the reviewers. Thus, well experienced reviewers with a long track record of providing reviews for DocEng provide the right balance to newcomers. Reviewer instructions as the reviewers to focus their reviews on key aspects that are considered important in DocEng papers. These include novelty and originality, clarity and cohesiveness of the submission as well as the relevance to the DocEng community. Such guidelines give the reviewer focus for their reviews and minimise the making of personal or unfounded judgements.
The adoption of the post-review comments also ensures that reviews that are unfairly critical or overly lenient are discussed among the peers before these are sent to the authors. This acts as a self-regulatory mechanism for the reviewers.
Finally, the program chair checks and reads all reviews and comments, making sure that all are of the appropriate tone and offer constructive criticism to the authors and, if necessary, engage in discussion with the reviewer to provide rephrasing of the review and even, if considered necessary, withdraw the review or ask for an opinion from a fourth reviewer.
Responsibilities
Of course, each role in the peer review process has responsibilities that must be taken seriously for the success of the peer-review.
The Reviewer
The reviewer must, first and foremost ensure that they allocate sufficient time to be able to carry out an effective review. This allocation of time must take place well within the deadlines given by the program chair/editor. Particularly for conferences, the program chair would have set the deadlines taking into account the strict timeframe required to have the proceedings published in time. This means that any deviations from the deadlines are likely to set the timeframe back. Unexpected circumstances can, and will occur and thus, it is important to always communicate with the program chair. If, for example, a review is in progress but will be submitted slightly after the deadline, then letting the program chair know of this will not result in frantic, last-minute searches for "emergency reviews". Likewise, if circumstances change between the time of accepting the reviewing role and the actual start of the review, then alerting the chair ensures that papers are reallocated to other reviewers in a timely manner. This is not only fair to the authors, but also to the reviewers stepping in.
It is also important to always act with academic integrity. If conflicts of interest exist, these should be flagged so that papers written by authors with whom you are close are not reviewed by yourselves. This removes the risk of fake reviews that is, instances where author groups agree to favourably review each other's works, or worse, their own works. If any such behaviour is suspected, this should be communicated!
It is also assumed that works submitted for review are sent to the reviewers in confidence. Thus, the works, whether they are accepted or rejected, should not be used for the personal gain of the reviewer. If the paper is accepted for publication, then this should be cited only after the proceedings are published. If the paper is rejected, any good ideas that the authors proposed should not be used as one's own!
The program chair
Among the responsibilities of the program chair is the need to ensure that a good program is presented to the conference attendees. This would include setting an appropriate call for papers and scheduling of the event timeline from the first call for papers to the final proceeding publication. But of relevance to the peer-review process include recruiting a suitable and diverse program committee and taking care that there are enough reviewers to cover the expected number of papers without unduly loading the reviewers. The program chair is also responsible for ensuring a suitable time frame for the reviews to take place, allowing for contingencies such as extending the paper submission deadline, and allowing time for discussions to take place.
With increased awareness of the possibilities of papers being generated through paper-mills, or AI generated papers, the program chair is then responsible for going through the papers before submitting for review to have a cursory overview of the papers and flag any obvious irregularities.
It is also the responsibility of the program chair to pair reviewers with papers in which they have expertise in, and to try, as much as possible, to match experienced reviewers with newer reviewers. It is also the responsibility of the program chair to ensure that conflicts of interest are avoided. These could be due to author groups belonging to the same institution, which are generally automatically flagged by the reviewing platform used. But it could also include recently collaborating authors. Such conflicts should be declared by the reviewers and the program chair needs to ensure that these are respected.
Finally, once reviews are submitted, it is the responsibility of the program chair to promote discussion, particularly if the reviewers reach conflicting recommendations. Since the program chair has an overarching view of all reviews, it is then within their responsibility to review all reviews to regulate the level of critique from different individuals on different papers to reach a decision about each paper. This decision should consider the overall program of the conference, keeping in mind the initial call for papers such that there is also a varied representation of all thematic areas of the conference.
Top 10 Things to Look for in a Review
1. Kindness
Frame your review how you would like it framed if it were your paper being reviewed. This will ensure that the authors can use your review as a learning experience. This, however, does not mean that you are not critical of the paper if criticism is necessary. You should not overdo it: be kind, not sappy!
2. Emphasise the Good
Most papers are going to be cited later for the single (occasionally two or more) biggest ideas they contain. You can use your position as a reviewer to highlight what you consider to be that single most influential idea that they expose in the paper. While it is important to help authors have a paper that is readable and improve all aspects as appropriate, your review could really help the authors to develop that core, differentiating contribution.
3. Background Research
It is important that you frame the review within the context of existing research. You may be assigned papers which, while you have the general expertise on the topic, are not directly aligned with your research projects. Look up recent works by searching the title/key words/key phrases on Google Scholar. Find the top 1-3 papers in the area and see if these are included in the background/discussions to verify that the paper is contributing to new knowledge.
4. Plagiarism Starts at Home
Sure, Wikipedia, ChatGPT, and other scientific communities exist, and it is important to check that the paper has not been plagiarised or AI generated. You can do this in various ways. You could find the rarest terms in the paper and search for them on Google Scholar to see what results are produced. You could also use your favourite plagiarism detection software (Grammarly, Turnitin, etc.).
5. Is it Reproducible?
Papers are published with the intention of sharing with others the research findings or ideas so that others may replicate and continue to advance knowledge. Thus, you should go into the reviewing intending to replicate the experiment being described. Assuming you had all the resources at hand, would you be able to repeat the experiment based on the instructions described?
6. Materials and Methods
Could the authors have used a different approach to carry out their experiment? Or have they included all the relevant literature? Your review should update the materials and methods mentioned in the paper with any of your own “improvements” if you feel that there are serious omissions. While you should not try to identify yourself, or push your own citations, the paper will be better without omissions. This of course also applies to suggestions of any relevant third-party materials and methods missing.
7. Are the Results Believable?
Your review should evaluate whether the experimental data supports the claims being made in the paper. Are the data credible? In addition to value (mean, median), is variability provided? Is it an “n=1” experiment? Has the experiment been set up for an analysis of variance (ANOVA) verification? Where parametric and nonparametric statistical tests used appropriately?
8. Discussion
In your discussion of the paper, try to include one or more observations of things the authors may have tried/included. Look for some sort of sensitivity analysis to give broader perspective. Check that there is no overfitting of the results. Look closely for tight coupling between the experiments and the discussion.
9. Adjacencies, Adjacencies, Adjacencies!
As in real estate, it’s all about location - how close is the work presented in the paper to other potential applications. Authors may not necessarily identify their perfect application at first, so by looking for other potential applications, you, as a reviewer, may add most to the author's contribution to science.
10. Will It Last?
Authors may be selective in what tests and results they report in their paper. As a reviewer, you will need to question what could potentially deprecate/obviate the reported findings? Have the authors employed Popper’s Falsification Principle in their experimental design, that is, have scenarios in which the proposed theories do not work been discussed? Results and discussions that only show support to the proposed approaches raise suspicion that authors are being selective in the evidence provided to support the theories presented.
Writing the Review
There is no single correct way for writing a review, but we can provide some tips which may assist with the review writing which work for us:
First read the paper in its entirety to obtain a feel of the paper, understand what the paper is about, the approach used, and results obtained. Once the first read-through is carried out, read the paper again this time with a greater eye on detail taking notes as you read, keeping the ten tips discussed above in mind.
Start the review by giving an outline of the basic contributions of the paper. This should be a few sentences describing the topic of the paper. Beginning with this helps the program chair categorise the paper and lets the author know that you have understood the paper.
Give your recommendation about the paper, providing justifications for your recommendations. These are your major comments about the paper and help the program chair, fellow reviewers and the authors understand the reasoning for your recommendation. This is also the time when you can provide feedback to the authors concerning clarity and soundness of their approach, interpretation of results, missing citations etc.
Include additional comments about the paper. These are minor comments and would include minor issues with spelling and/or grammar, suggestions for changes to figures and tables etc.
Check that your review addresses all the points that are suggested by the program chair as deciding factors for your review.
Once your review is written, take a step back and re-read your review. Check it for errors and clarity.
Do not be a Reviewer #2!
We have all been a victim of an infamous Reviewer #2 at some point or other, and the flaws in the peer-review process will remain. We iterate that a bad review is not necessarily a critical review, while an overly saccharine review is definitely a bad review! Criticism, when set at the appropriate tone, will provide authors with a teaching experience that has the potential to improve the research, analytical and reporting skills of the authors. If we all make the effort to execute proper peer reviews, then we may begin by providing these learning experiences in our research areas, bettering our research communities.
This content of this report was presented as a tutorial with the same title at DocEng'23 which was held in Limerick on Tuesday 22nd August 2023. Details of the tutorial can be found at: https://dl.acm.org/doi/abs/10.1145/3573128.3607810
Steve Simske is a member of the Walter Scott Jr. College of Engineering at the Colorado State University and is the Chair of the DocEng Steering Committee while Alexandra Bonnici is a member of the Department of Systems and Control Engineering at the University of Malta and served as the program chair at DocEng'23.