Peer Review – Institutional Hypocrisy and Author Ambivalence

You will forgive my ‘deformation professionnelle’ by returning again and again to the subject of peer reviewing.

Most law faculties in most jurisdictions have been moving toward the use of various modes of quantitative indicators in the process of appointment and promotion of their faculty. Even the United States, which until recently has been a blessed exception in this regard, is now taking first steps down that slope – prompted by the decision of the US News and World to include faculty ‘productivity’ in its rankings.

Make no mistake, there is a place for such indicators in the overall assessment of a legal scholar. In its extreme form – which is hardly exceptional – one counts the number of conferences, the role played in those conferences, the number of papers given, articles written, citations, and the like. And then ‘points are given’. So many points are needed for this, and so many points for that. I have seen the European Research Council and the Commission of the European Union under its various (laudable) research support programmes (Horizon 2020 and its antecedents and progeny) follow the same points method.

The rationale is not without merit: an attempt to move away from a combination of the odious ‘old boy network’ where appointments were determined according to who you knew and who supported you (my Baron is more powerful than your Baron) and away from subjective judgments of quality towards some objective methodology in the interest of fairness and academic excellence. Hence a ‘market approach’. Let quality be decided not by our judgment but the judgment indicated by the ‘quality’ of the journal in which you publish, by the number of citations, etc. Coupled with the attempt at qualitative objectivity – the desire for which is understandable and serious – there has been a shift to a ruinous attention to quantity of publications (publication being the Alpha and Omega of academic excellence, as if teaching or all aspects of academic citizenship, such as committee work, peer reviewing, tenure review, etc. count for nothing, or near nothing.)

When deans review the end of year report on the ‘productivity’ of their faculty, they smile at a faculty member who reports, say, six publications in the last year and frown at the member who reports only, say, one. They smile at the faculty member who reports attendance at, say, half a dozen conferences and a few workshops, especially if they served as moderator/chair, commentator, President of a Session or even a Keynote (Douze Points!) A festival of points ensues. And the poor faculty member who only attended one conference or perhaps none will hang his or her head in shame with their solitary point – and with potentially very serious career consequences. And yet that solitary article may have been truly brilliant and of an altogether higher quality than the conference, edited book ephemera. Here the ‘market’ that is in play is not the internal academic market, but the real market of external evaluation by those who control the purse, and they, too, want some ‘objective’ indicators, so they count. It would not surprise me if eventually the ‘rankings’ become not a once a year affair but, like in tennis or snooker and now even in soccer, there will be a running ranking where these quantitative indicators are aggregated in real time and we will learn that faculty x moved last month from 13 to 11.

But market failure is endemic and anomalies abound – here are but a couple of examples. A scholar whose article is abundantly cited negatively for, say, poor methodology, will score higher than one whose paper is cited scarcely but eulogistically. A scholar who, as mentioned above, eschews the conference circuit (and circus) and spends, say, two years on writing a truly fundamental article will score lower than his or her colleague who attends endless conferences, delivers endless ephemeral papers that then get published. And books are another area of footnote market failure. Many of the ‘tracking agencies’ do not reference footnotes citing books, so a serious scholarly book might produce fewer ‘points’ than its value merits.

I find the reliance on footnotes particularly destructive. My long experience as an editor of two learned journals has taught me that footnote counts are tremendously fickle and unreliable. Certain subjects by their nature attract more attention than others, creating a disturbing incentive when young scholars determine their research agenda. Older, more established scholars attract more footnotes, even if they are just selling the same old goods, whereas young unknown scholars will be overlooked in the footnote game.

But this is well-covered terrain, even in these pages and no more need be said.

As noted, in an attempt to insert a qualitative dimension in the quantitative counting, publications are weighted as more significant – and in some jurisdictions this is a sine qua non for being counted at all – if published in a peer-reviewed journal, and among these most weight is given to journals that ‘rank’ in the ‘top tier’ of some journal ranking. No sour grapes here – EJIL usually finds itself in such top tiers.

In effect, faculties are, at least in some important measure, outsourcing the quality control of the work of their scholars to peer-reviewed journals. (I suppose I am biased since in the process of appointment and promotion in my own faculty almost exclusive attention is given to a careful reading of the work by the faculty itself and the external indicators play a minor role if at all. I fear this is about to change.)

Be this as it may, we, editors of peer-reviewed journals, understand the ways of the world as it is and take this responsibility with utmost seriousness, not only to guarantee our readers that only articles of high or very high quality are published but also because we are aware of our responsibility in the appointment/promotion cycle.

We select peer reviewers with care and after discussion, based on our knowledge of their work, their standing in the field, their proximity to the subject of the article, and so on. We believe that being selected as a peer reviewer by a quality journal is no less a mark of recognition and distinction than, say, delivering a paper in many a conference, to give but one example. A good peer review requires application, careful reading, exercise of judgment and, I fear, quite a bit of work.

I do not recall a single instance of an article being published in EJIL without some revision advised or required. There is nothing so good that cannot be made better. A good peer review can run to several pages, providing a suggested roadmap for the revision of an article. It often involves several rounds among peer reviewer, editors and author. On most occasions we receive warm thanks from the author when the process is concluded – recognizing that their good article was made even better as a result of the process.

Peer reviewing is not only in some ways a measure of recognition of quality and distinction of the peer reviewer, but also an act of high academic citizenship, which, as noted, does not only serve the author and the journal but serves the ecosystem of academic appointments and promotion.

This is where the institutional hypocrisy comes into play. It is a service which, in an ironic paradox, receives no institutional recognition in the processes of appointment and promotion. Faculties insist on publication in peer-reviewed journals. But the real ‘heroes’ in this process are not the journals, but the peer reviewers. I have heard more than once from colleagues who are reticent to undertake peer reviewing because it involves a lot of work (it does, when done well – take a look at a recent Editorial ‘Best Practice – Writing a Peer-Review Report’) and in a world which counts (and gives points) there are no points (real, or so to speak) for peer reviewing.

To me the point, excuse the pun, is obvious. I think that peer reviewing should become a standard feature in a candidate’s file. And if faculties rely on peer reviewing by the top-ranked journals, they should find a way to give incentives to an institution on which they rely: peer reviewing should be acknowledged and rewarded like other facets of academic achievement. As part of the institutional culture, peer reviewing should be viewed as an indispensable norm of good academic citizenship. At EJIL and ICON we are considering issuing some form of ‘certificate’ to our peer reviewers in the hope that this will become a common practice and we will encourage scholars to submit them in their application/promotion files as well as in annual ‘productivity’ reports.

What then of author ambivalence? The practice of peer reviewing differs from jurisdiction to jurisdiction, so occasionally there are crossed wires that result from cultural differences. These are relatively easy to sort out. For the most part, when authors submit an article to a peer-reviewed journal they understand the rules of the game. As an empirical matter, the frustration and ambivalence are rooted in two principal factors:

• The time element: peer review can add anywhere from three to six months to the processing of an article. And then, to add ‘insult to injury’, the revised and approved article must take its place in the publication pipeline, adding several more months waiting time. The opportunity cost is particularly high if the result of the peer review is a rejection.
• Substantive disagreement and frustration with the content of the peer review: s/he simply did not understand or evaluate correctly my article.

We are acutely aware of these issues and have taken several measures to mitigate the frustration. As regards the temporal element, we have adopted two policy changes at the beginning and the end of the process. We no longer require exclusivity in the initial submission of an article to EJIL. We guarantee, and in almost all cases honour this guarantee, to give an initial screening decision – whether or not the article will go to peer review – within six weeks of submission. If we decide not to send an article to peer review (and there can be many ‘curatorial’ as well as quality reasons for this, such as ‘we have something on this topic already in the pipeline’), the author will not have wasted precious time since the article will be making its way through the process of any other journal to which it was sent. If we decide to peer review, the author is informed and at that point we do still insist on exclusivity. So much editorial work goes into the peer-review process that it would be unacceptable to do all that and then find that an author just skipped boat.

We are now moving to a system of pre-publication, so articles that are accepted following the peer-review process will appear online ahead of the formal issue in which they will eventually be published. Additionally, with an eye to the appointment/promotion process, we are always happy to send authors a formal letter affirming acceptance of their article for publication. This takes care of that problem in almost all cases.

Both these measures seriously attenuate the time factor frustration, but we are not willing to cut corners in the actual process of peer review. And, as mentioned above, most authors at the end of the process express their gratitude.

Peer reviewers are not infallible, nor are Editors in Chief. When the revisions required of an article are very substantial, we typically invite the author to submit their reactions and indicate the changes they plan to introduce to the article in the light of the peer reviews. Authors regularly object to this or that point in the peer review and we regularly accept such reservations. Oftentimes we will point out to authors that if a serious peer reviewer failed to understand a point, it might at least be worth considering whether the writing can be clearer on this or that point so as to avoid the same misunderstanding by eventual readers of the piece. This dialogical approach ensures that even misunderstandings can turn out to be productive. But there is no getting away from the fact that peer reviewing does add at a minimum several months to the publication process.

It is a fact of life that most articles winding their way through the peer-review process exist in one form or another online as, say, an SSRN paper. We have become relaxed about this, provided the final version accepted for publication follows our copyright rules – which I believe are among the most generous in the field.

However, beyond these pragmatic considerations, I attribute author ambivalence towards peer review to a common misconception – that peer review is primarily about judging: good or bad, publishable or not.

In fact, statistically, for the most part, submissions that pass screening and go to peer review result in required revisions rather than outright rejection. We remind our peer reviewers again and again that apart from their judgment – accept or not – they should provide a detailed roadmap to help the author attend to any criticism they have. In effect, the greatest service that peer reviewers provide is not the judgment, but the selfless help to colleagues in order to make their writing the best it can be before publication. I think this point is not always sufficiently appreciated and peer reviewing is regarded as a necessary evil imposed by the powers that be for the purpose of career advancement. That is a pity.

Finally, even when the peer review results in rejection, we attempt to provide the author with a reasoned report, which at a minimum may induce the author to rethink the article before submitting elsewhere. We do not lightly reject an article that has passed our screening and gone to peer review, and we do this only when we are convinced that mere revisions are not viable.