magnify
Home Editorials Peer Review in Crisis

Peer Review in Crisis

Published on July 12, 2012        Author: 

EJIL and its sister publication, I-CON are peer-reviewed journals. This is a counter-cultural posture in an age which celebrates, for some very good reasons (and some less admirable), the freedom that self-publication on the internet provides. Our own very successful Blog, EJILTalk!, is an example of a highly interesting and useful form of self-publication and I-CONnect will be launched soon. There are surely others like ours. SSRN is a more ambiguous example, but even there, there are some diamonds in the rough, if you have the patience to do some heavy-duty prospecting and sifting. Be that as it may, SSRN is not just part of contemporary academic culture; it is a defining part, both reflective and constitutive.

There is a place, we maintain, for discernment in publication, including external referents. There are some weeks where the (electronic) mailman (in the form of ScholarOne) sends my way one or even more articles per day for both EJIL and I-CON. We need to select, not simply because the economy of a journal dictates such, but because we try to give our readers a certain guarantee of quality, even excellence. We know, too, that in many countries, publication in a selective, peer-reviewed journal plays an important role in appointment, promotion and tenure.

At the heart of such a system is, indeed, peer review. This institution is in serious crisis, which is evident in the functioning of both journals. I have discussed the issue with other Editors in other journals and the situation is the same elsewhere. I am, thus, taking the extraordinary step of publishing a similar editorial in both EJIL and I-CON.

At EJIL (and I-CON) we try to practise double-blind peer review: in principle, the reviewer should not know the identity of the author, and the author, obviously, is not privy to the identity of the reviewer. The double-blind principle is not always achievable. We do not have the resources to scour each and every article that goes out to review and excise from it all tell-tale signs, notably footnotes of the ‘see-my-treatment in…’. Some authors have a distinct voice which is impossible to conceal. And, as I explained in greater length in an earlier Editorial (‘Demystifying the EJIL Selection and Editorial Process’, at 22 EJIL (2011)), since we like each piece we publish to have had critical scrutiny by at least two sets of eyes, oftentimes one of the peers is myself; obviously I am aware of the identity of the author. In that case the double-blind principle will apply only to one of the reviews.

We give considerable thought to the selection of ‘peers’. We look for people who have expertise in the field and whose own publications meet our yardstick of excellence. We make liberal use of our own Scientific Advisory Board and Members of the Editorial Board. But given the volume and diversity of submissions we receive, even after our in-house screening which reduces the numbers considerably, we need to venture outside and turn to the legal academic community at large.

Why crisis? Simple enough: first, the difficulty of finding willing peer reviewers. Peer reviewing is a selfless task. Though we have taken to sending a little gift to external reviewers in the form of an Amazon Token, peer reviewing is an act of academic citizenship which demands sacrifice in time and mental energy. Sometimes our turn-down rate exceeds 50 per cent. Not infrequently we will get a refusal from the first and second and third reviewer to whom we turn, whilst the hapless and unsuspecting author is fuming at the length of time it takes EJIL to reach a decision. I find it particularly galling, yes – galling, when a published author in EJIL or I-CON refuses to review, having himself or herself previously enjoyed the fruit of the system. If ever there was a case of spitting into the plate whence one eats.

Second, not infrequently I receive reviews which are perfunctory, conclusory, poorly reasoned and hence appear arbitrary, effectively amounting to an ‘it’s bad because I say it is’ or ‘it is good because I liked it’. This is not only unhelpful to me as Editor, but raises doubts in my mind whether the reviewer has taken time to read and ponder the article carefully. In such cases, and the number is not trivial, the process has to begin again. Since I have been with EJIL throughout its life (though only recently as Editor-in-Chief) and with I-CON for some years now, I can also point quite distinctly to a deterioration in the situation. One can only speculate as regards the reason: Perhaps a proliferation of Journals with a corresponding increase in the number of times one is asked to review? Perhaps a change in culture where the very institution is eclipsed by self-publication, and  peer reviewing has fallen outside the basket of academic citizenship virtues? Perhaps everyone is more busy writing?

And yet, we really cannot do without peer review. We can rely on our ‘own resources’ – the gallant members of our Scientific Advisory Board and other members of the Board of Editors only so much.

What is to be done?

  • We need, perhaps, to revisit the structure of incentives. When one is asked to peer review a prospective book, one is typically offered some free or discounted books by the publisher commissioning the review. Let’s see if we can work something out with OUP.
  • We will be shortly opening a Register of potential reviewers, a roster which we can then dip into.
  • In our letter of acceptance to authors who have submitted an article to I-CON or EJIL we plan to prominently emphasize the gratitude to the anonymous peer reviewer and emphasize the moral duty to commit to this duty of academic citizenship.

 We welcome other suggestions from our authors and readers.

Print Friendly
Filed under: Editorials, EJIL, Journals
 

5 Responses

  1. Zvi Ziegler

    I understand the predicament, and appreciate the analysis .

    I believe that in a double-blind peer review a brief rejection/acceptance is an option which should not be derided. The reviewer, after reading the paper, may form a quick impression, provided she/he is an expert in the field. Sometimes the impression is positive, and there are improvements or modifications that may be required as a condition for acceptance., and then the reviewer may ask for these. At other times, it may be clear to the reviewer that the paper is worthy of publication as is , or that is beyond repair. In such cases , the added burden of having to explain and argue , is superfluous . It is the reason for the high rate of refusals to engage in reviewing .

    In summary –
    1. An expanded register of reviewers is an excellent idea.
    2. The advisory board’s main task would be to choose the reviewers from the pool – and making sure that she/he is an expert in the field of the paper to be reviewed.
    3. The reviewer would be explicitly given the option of accepting/rejecting the paper without an explanation of her/his reasons.
    4. An active reviewer – one who writes more than a certain numbers
    of reviews (say five a year) will be acknowledged as such. This will be counted as an “academic activity” – in promotions etc., and thus can serve as an incentive.

    I think these suggestions will reduce the percentage of refusals , and will shorten the time to reach a decision re acceptance., two of the points that justly bothered you

  2. Joost

    I think there might not be enough willing reviewers because they are too busy writing commissioned articles..

  3. maria-ruxandra stoicescu

    Perhaps the whole system needs an overhaul. It sounds illogical to me that one would want to remove an author’s voice, see the “some authors have a distinct voice it is impossible to conceal”- really? Then how is original research ever going to emerge, if everyone is looking to conceal their authenticity? maybe peer reviewers are tired of reading pieces that end up sounding like one another simply because authors write with peer reviewers in mind rather than with expressing their ideas. This is not to say remove peer reviewing, but about make it more rewarding intellectually for both writers and reviewers. A discussion rather than a double blinded firing squad. Less universal standards and more listening to what research and authors have to say. Peer reviewers end up being enforcers of a system they loath and reinforce at the same time. It’s hard for refreshing pieces of research to get through. People are only tired and busy when what they do is seen as a closed circle, there is always energy, time and enthusiasm where there is openness.

  4. Totaly agree with what you call “discernment in publication”. Thanks to this requirement, your review and blogs offer serious and useful informations for both searchers and professionnals engaged in the daily practice of law.
    You can also visit this new francophone website on IHL:
    http://www.larevuedepressejuridique.org
    Best regards,

    J.

  5. Joost

    Maria-Ruxandra, your point is well taken.

    The Poet once said: “Heartless Western Culture has judgmentally excluded different voices based on cruelly judgmental and archaic criteria; decent people understand that modern Academia is a way for everyone to participate in legal analysis no matter what their capacity.”

    Best,

    Joost