Best Practice – Writing a Peer-Review Report

The importance of peer review has, if anything, increased in recent times. The enthrallment of current academia with ‘objective’ quantitative measures in the processes of selection, promotion and evaluation of academic performance has put a premium on publication in ‘peer-reviewed’ journals. Instead of a faculty reading carefully the work and making up its own mind as to its quality, they will outsource such to two anonymous peer reviewers. Also, in the face of the avalanche of self-publication in outlets such as SSRN (valuable in and of itself) and the like, peer review may help the discerning reader navigate these channels, thereby providing some guarantee of excellence.

Yet this importance is often not matched by the practice of peer review. The rate of refusal to peer review is as high as 50 per cent – oftentimes by authors who themselves have published in, and benefited from, peer-reviewed journals. Authors who publish in EJIL and I.CON undertake to peer review for our journals, an undertaking not always honoured. Of course, there is only so much peer reviewing that one can do and we understand when we receive a request to beg off with a promise to do it on some other occasion.

Then there is the problem of tardiness. Four to six weeks is a reasonable time to expect a peer-review report to come in. Frequently, to our and our authors’ frustration it can be as long as 24 weeks, after a slew of ‘gentle’ and somewhat less gentle reminders.

And then there is the question of the quality of the review, oftentimes perfunctory and hardly helpful.

So here are some guidelines to this act of high academic citizenship.

There is a common misconception that the most important thing the Editors want is a judgment: publish or do not publish. Of course we are interested in that final judgment. We rarely, if ever, will publish an article where both peer reviewers have recommended rejection. But it should be remembered that all journals engage in initial screening, picking out the articles that will be sent to peer review. Articles may be screened out for a variety of reasons, such as subject matter interest, pipeline management (too many articles on the same topic) but also, of course, for quality. The editorial team will not send out to peer review articles that they expect will be rejected – that is not a clever way to manage this scarce resource. The result is that articles sent to peer review are those considered potentially publishable which, in turn, means that the most common outcome of the process will be a double ‘revise and resubmit’ (R&R), in borderline cases one R&R and one rejection (R) and rarely two Rs.

Even when the recommendation is a straight A (acceptance, excuse the pun) good peer reviewing will still provide the Editors with a reasonably detailed evaluation of the piece, explaining their view of its quality and the original contribution it makes and, operating on the principle that there is nothing that is so good that it cannot be improved, providing abundant suggestions and recommendations to the author. The evaluation for the Editors is particularly valuable in those cases where there is a divergence of opinion among the peer reviewers. Explaining the originality and importance of the articles is key in a positive peer review because the Editors, knowledgeable as they may be, cannot be masters of all specialties in the general field and, whereas they can discern good or bad writing, powerful or weak reasoning and the like, will often be less confident in assessing originality and importance in areas not their own.

When the evaluation is negative it is even more imperative to invest in the rejection report, both for the benefit of the Editors as well as for the benefit of the author. If, for example, the claim is that the piece is not original, it should be accompanied by a couple of references to work that substantiates that opinion. If the negative judgment goes to quality rather than, or in addition to originality, for instance if it is poorly reasoned, contains lapses in argument and the like, once again this should be spelled out in the report.

The single-paragraph peer-review report which one sees from time to time – ‘This is an unoriginal, poorly reasoned and badly written article: Reject!’ – is singularly unhelpful as well as lacking credibility. It is unlikely that the Editors would send out such a piece to peer review, and imagine yourself, as an author, receiving such an evaluation.

Here are some pitfalls in the judgmental function of peer reviewing.

One of the most common pitfalls is the confusion between ‘I don’t agree, the author is wrong’ and ‘this is a bad article’. I will grant you that the line between the two can be fuzzy, but you grant me that there is, nonetheless, a difference between these two categories of judgment. And since we always seek specialists in the field covered by the article submitted, this inadvertent danger is enhanced since the peer reviewer has a stake in the field, in positions taken and the like. The only remedy is awareness of this distinction and self-awareness in the process of peer reviewing.

Another pitfall is ideological bias in the peer review. We notice this from time to time when we receive divergent evaluations of quality which track the ideological disposition of the peer reviewers. The remedy in this case is similar: awareness of the potential problem, self-awareness, and a sense of intellectual integrity.

The most common pitfall is … being perfunctory. A quick read, a quick report. A careful read, some reflection and a thoughtful non-hurried report is the Gold Standard.  

For the reasons explained above Revise and Resubmit is the most common judgment and the report requires a little extra work. In the first place, the articulation of the defects that prevent a straight A should, somewhat paradoxically, be more detailed and expansive than those in a Rejection report, since here one wants to list not only those defects that are lethal to publication, but also non-lethal defects correction of which would improve and enhance the eventual published article.

Critical in an R&R report is the ‘road map’ approach. The author should not only understand the weaknesses but should understand perfectly what needs to be done so that, if performed to an adequate standard, the revised article is likely to be accepted on resubmission. In drafting the R&R report, the peer reviewer should keep in mind this road map approach. Editors will often send the R&R report to the author, asking for reactions to the recommendations, an indication whether they are in agreement with the peer review and an indication of how they plan to revise the piece. The clearer the road map, the better this process unfolds.

Peer reviewers are typically asked whether they would be willing to review the revised piece. It always shocks me a bit when the No box is ticked. It is like a job left half done. Who better than the peer reviewer herself or himself to evaluate whether the revision is satisfactory? But, be that as it may, if the ‘not willing to review the revision’ box is ticked, all the more important to have a very clear and elaborate road map so that the Editors themselves can better evaluate the revision.

Good peer reviewing probably requires somewhat more effort than preparing to comment on a paper at a conference. Yet it does not give the same exposure and, scandalously, is not taken into consideration in many places when evaluating the file of academics. These days people list in their CV their blog entries. I believe that listing peer reviewing should become standard practice. If deans and the profession place, as they do, so much weight on publication in peer-reviewed journals, surely it should mean something as regards reputation in the field that someone has been trusted to undertake peer reviews by those very journals. This thankless act of high academic citizenship should be valued.

As a token of our deep gratitude to our colleagues in the international law field who accept our invitation to review for EJIL, we offer a free online subscription to the Journal for one year and, in addition, OUP provides a 30 per cent discount on book purchases.

Furthermore, as a start towards confirming broader recognition of the importance and value of peer reviewing to our Journal and to scholarly writing in general, we plan to institute a special prize each year to the referee of an outstanding review. In particular, we hope it will be a signal to deans and other academic authorities that peer reviewing should be considered as a meaningful element in assessing both the scholarly impact and academic citizenship in the context of appointments, renewals and promotions.