The Art of Providing Good Reviews.
Steve Simske, Alexandra Bonnici
As of 2022, over 5.4 million academic articles are published each year. These include short surveys, reviews, conference proceedings and journals. The gatekeeper that ensures the quality of these articles are peer-reviews: a tried-and-tested way through which other expert academics decide on the relevance and contribution of the article to the general knowledge in the subject area. And while we, as authors, may not particularly be overjoyed at receiving a negative or critical review, there is a difference between a review that offers a critical and meaningful review and one that is just a bad review: the reviews that are one-liners, offering no help, to the downright snarky ones that leave authors disheartened with the paper publishing process.
If there are reviewers who provide good reviews and others who provide less than stellar ones, then how do conference chairs go about selecting a suitable group of reviewers?
PhD comics neatly sum up how reviewers are selected. Only, this is not the bad thing that it is made out to be in the comic strip. A program committee is made up from people from within the community and recruiting new reviewers from the previous conferences is one way to keep rejuvenating the programme committee. But there is some truth in the comic strip. How does one provide a good review if one does not have, or consider themselves to have sufficient experiences? Can we train young researchers in conducting effective peer-reviews?
Note that in this report, we are not advocating for non-critical reviews. If a paper has flaws, it is important that these are highlighted so that authors may address them. What we would like to advocate for is for reviews to be a teaching experience.
Example 1
Even the authors could not clearly explain in the conclusion or in the body what are new results. The article is not suitable even for a poster, since it is in the very early stage of the investigation. Numerical experiments are not of sufficient interest for a research article.
Example 2
The authors apply some standard approach to some unavailable data set and produce some numbers. Where is novelty? This is just a report on some work and is not a suitable paper for this conference!
Example 3
Great paper; recommend acceptance.
Reviews such as those of Examples 1 and 2, while potentially correct within the context of the paper, do not really explain to the authors what could have been done better. The reviews belittle the authors' work, and both may be extremely disheartening for the authors receiving the review. The third example, while positive, is also a bad review! It offers no insight on what makes the paper such a great one, making it difficult for the program chair to understand the justification behind this recommendation. It is these types of reviews that we feel must be stopped.
The Peer Review
The peer review is a quality control mechanism that subjects an article to the scrutiny of independent and impartial experts in the respective fields. These experts, referred to as the peers or reviewers, will judge the article for its content, methodology, originality, and the overall significance of the work. The reviewers will also provide feedback and recommendations to the authors to help the authors improve their work.
The peer review process is therefore a quality assurance process, subjecting the authors’ work to expert evaluation that exposes errors, inaccuracies or deficient that must be resolved before the article is presented for general dissemination. This, in turn, ensures that the research published is both reliable and credible.
The scrutiny of the reviewers ensure that the methodology adopted are theoretically appropriate and technologically sound, hence validating the findings reported in the paper. Through the peer review, we have the assurance that results are robust and interpreted appropriately.
The constructive criticism provide by the reviewers provides the authors with feedback that helps the authors to clarify arguments or improve methodology with the scope of providing a more impactful and meaningful contribution to knowledge.
In short, the peer review engages the community to build upon each other’s knowledge and experiences to collectively advance knowledge in a specific area. This requires a community of reviewers that are objective, impartial and genuine in their intent to develop the research area. Providing this kind of feedback takes time and effort, but done properly, the peer-review process offers several benefits to authors, the reviewers and the research community.
Benefit to Authors
For authors, the peer review process gives the assurance of improved quality of the contribution and research. Even if the paper is rejected, good reviews will provide the authors with invaluable insights on how the paper, and their research, may be improved for the next submission of the paper. This helps the authors submit stronger contributions.
The peer review process will also result in publications that can be trusted. Knowing that a paper would have passed through the scrutiny of experts assures researchers reading the paper that the work meets certain standards. This lends credibility and validation to the authors’ work. This credibility implies that peer-reviewed works are more likely to be read and subsequently cited, thereby increasing the visibility and recognition of the authors within the academic community. The exposure and recognition garnered would indirectly increase the opportunities for collaborations, exposing the authors’ work to other researchers in the academic and professional community.
Benefit to Reviewers
For reviewers, the peer-review process means that the reviewers have a direct contribution to the field, giving the individual the opportunity to have an active role in shaping the quality and direction of research in the field. Through constructive feedback the reviewers are helping authors improve their work and hence, contribute to the advancement of knowledge. In providing good quality reviews, the reviewer must keep abreast with the current state-of-the-art and good practices in the field. Through providing the review, the reviewers, particularly young reviewers, are enhancing their critical thinking and analytical skills to evaluate the strengths, weaknesses and contributions of the works being evaluated. This results in personal growth of the reviewer as an academic professional.
Although the PhD Comics strip made fun of asking the previous year authors to act as reviewers, this is generally a sign that the individual’s academic contributions are being recognised. Serving as a peer reviewer is then a sign that the expertise and reputation of the individual are being appreciated by the academic community.
Benefit to the Academic Community
For the academic community, and in particular, program chairs or journal editors, the peer-review process ensures that decisions are reached based on consensus among an admittingly small sample of the research community. The peer-review then establishes a small communication and discussion link between field experts to address controversies and reach consensus, assisting the editor or program chairs in determining whether the submitted work should be accepted or otherwise.
The Review Process
Responsibilities
Top 10 Things to Look for in a Review
Writing the Review
Do not be a Reviewer #2!
We have all been a victim of an infamous Reviewer #2 at some point or other, and the flaws in the peer-review process will remain. We iterate that a bad review is not necessarily a critical review, while an overly saccharine review is definitely a bad review! Criticism, when set at the appropriate tone, will provide authors with a teaching experience that has the potential to improve the research, analytical and reporting skills of the authors. If we all make the effort to execute proper peer reviews, then we may begin by providing these learning experiences in our research areas, bettering our research communities.
This content of this report was presented as a tutorial with the same title at DocEng'23 which was held in Limerick on Tuesday 22nd August 2023. Details of the tutorial can be found at: https://dl.acm.org/doi/abs/10.1145/3573128.3607810
Steve Simske is a member of the Walter Scott Jr. College of Engineering at the Colorado State University and is the Chair of the DocEng Steering Committee while Alexandra Bonnici is a member of the Department of Systems and Control Engineering at the University of Malta and served as the program chair at DocEng'23.