“Refereeing Template”: A guide to writing an effective peer review

We offer here a “Refereeing Template” to guide a more constructive and systematic peer review process. This template provides the following algorithm for constructing a peer review: (i) list the key claims of the manuscript; (ii) determine if the data (and interpretations of the data) support said claims; and, only as a last step, (iii) offer an opinion as to whether the supported claims deserve publication in the targeted journal. The Refereeing Template is designed to improve the quality and clarity of reviews by emphasizing claims, arguments backed by supporting evidence, as the basis for evaluating the quality of a manuscript, and translating that evaluation into an assessment of impact and significance.

Peer review is a critical component of the scientific process, and serves as a primary guardrail in maintaining the quality of the scientific literature. The peer reviewer is typically tasked with evaluating the quality of a manuscript and providing an assessment of the novelty, significance, and breadth of appeal of the manuscript. However, most reviewer guidelines tacitly assume that both of these tasks can be carried out in tandem, and provide little guidance on how, if at all, deliberation on the former impacts the latter.
In order to guide a more objective, constructive and systematic review process, we offer here a "Refereeing Template". This template serves to emphasize "claims", arguments backed by supporting evidence, as the basis for evaluating quality and translating that evaluation into an assessment of impact and significance. The template provides the following algorithm for constructing a peer review: (i) articulate the key claims of the manuscript; (ii) determine if the data (and interpretations of the data) support said claims; and, only as a last step, (iii) offer an opinion as to whether the supported claims deserve publication in the targeted journal.

Refereeing Template
Step 1. Identify the primary "claims" of the manuscript ○ Enumerate the primary claims made by the authors. It is these claims that should be used to measure the impact of the study on the targeted community.

○
If the claims are ambiguous, the reviewer may elect to base their review on their best estimation of the claims (so enumerated in the review), or to recommend the claims be clarified before further evaluation.
Step 2. Assess whether the data support the claims ○ Have the experiments been fully described, and is the experimental data presented in full, without apparent exclusions? ○ Were the experiments comprehensive and adequately suited for testing hypotheses that underpin the primary claims of the work? ○ Are the claims adequately supported by the data presented? ○ State, where needed, how claims could be modified/restricted to bring them in line with the data. ○ State what experiments are needed to better support tenuous claims, as well as secondary claims.
Step 3. Assess whether the supported claims warrant publication in the journal ○ Several questions can be used to guide the evaluation of the impact of the claims: i) Do the claims differ from prevailing thinking? ii) Do the claims clarify an existing ambiguity? iii) Do the claims enable new avenues for exploration? ○ For those claims that have partial support, one can provide a differential assessment based on the possible outcomes of the proposed additional experiments. ○ Assess whether the scholarly presentation is appropriate for publication.

2/3
The Refereeing Template guides the reviewer to objectively evaluate the scientific quality of the work prior to offering a subjective assessment of whether the work is appropriate for the journal and target audience. This sequence focuses the energy of the reviewer on the objective aspect of the review first and foremost.
A productive dialogue about the validity of the claims and their impact can only occur after the author and reviewer have a common understanding of what the claims actually are. A reviewer's articulation of their understanding of the central claims ( Step 1) provides context for the remainder of the evaluation. Authors would then receive more concrete information about how to improve their work by clarifying or modifying their claims and/or conducting additional experiments to better support their stated claims.
The identification of the key claims provides a basis for the reviewer to assess quality (Step 2) by envisioning what evidence is required and whether the experimental data sufficiently supports the key claims. This sequence is intended to foster a more productive dialogue with all stakeholders -authors, reviewers, and editors -that focuses on the most critical pieces of supporting evidence and critical controls, rather than ancillary pieces of data that are less consequential to the central claims.
After conducting Steps 1 and 2, the reviewer is better positioned to consider the more subjective question of "Do the claims in the work, weighted by how well they are supported, have sufficient impact to meet the standards of the journal?" By expounding on this topic only after the identification and evaluation of claims, editors and authors will gain a clearer understanding of the logic applied by the reviewer and the reasons for their ultimate decision. This structure should enable a more meaningful scientific debate among the authors, reviewers, and editors.
The Refereeing Template is designed to improve the quality and clarity of reviews, and to emphasize deliberate empirical reasoning that reflects the ideals of the scientific method.