More Ideas and Less CV?

Various EJIL editorials have highlighted some of the dangers to the integrity of academic life of excessive reliance on quantitative indicators in assessing the excellence or otherwise of scholars in the different stages of an academic career, be it selection to a Post Doc programme, initial appointment as faculty or subsequent promotion.

Why Not to Rely Excessively on Numbers?

‘Productivity’ measured by the number of articles published puts immense pressure on authors, with the risk of sacrificing quality for quantity. The assessment method can lead people to publish essentially the same idea over and over again, with only variations in the packaging. As a journal, we also encounter this phenomenon when we receive articles based on research projects that have been ‘salami-sliced’ into the ‘lowest publishable unit’ (to borrow words from a valued peer reviewer); other parts of the research project are then published elsewhere. To the reader, however, the whole would be so much more valuable than the sum of the slimmed and trimmed salami slices.

‘Impact’ measured by the number of citations risks penalizing colleagues who write on important but non-mainstream themes. At EJIL, we are aware that some excellent articles which received glowing peer review may attract fewer citations only because they focus on off-beat topics, whereas less profound or original pieces may get many more citations simply because the field of writing is crowded and excitable. One can add to this the irony that a piece of scholarship may receive many citations precisely because it says something outrageous or ridiculous, as it then becomes an obvious go-to foil, thereby boosting the author’s H Index. As evidence of our not caring about this numerical factor, we have asked OUP to remove our impact factor from EJIL’s website.

Going Qualitative: Why Not to Rely Excessively on Curricula Vitae

In many circles of legal academia, we have thankfully not gone fully quantitative (yet). But qualitative assessment is often superficial, too. Given the amount of work involved, it is understandable that some selection and promotion decisions are made on the basis of CVs, without serious engagement with and deliberation about the scholarship itself. This is very common in selections for Post Doc programmes. Examine your own institutional practice: how often is the applicant’s doctorate actually read? One pernicious result of the lack of engagement with theses is the pressure on doctoral students to take time out of their doctoral research to produce articles so that their CV looks more impressive. You might have the most brilliant of dissertations, but absent additional articles the chances of winning a post-doctoral position will be seriously compromised. It is not uncommon for a similar practice to exist in assessing applications for academic positions, especially at entry level. A variant of such occurs when committees assess where an applicant has studied, worked, or published instead of engaging with what they have written. The university, employer, journal or publisher is taken as a proxy for quality. It saves the committee precious time, but also outsources the quality assessment.

This happens perhaps even more frequently at a more senior level, where people get recognition for … getting recognition. Once one has been given a prize by one professional society, another professional society offers its membership because of one’s fame for receiving the prize; on the basis of membership of that society, one gets nominated for other elite institutions, membership of which then becomes the ground for future promotions and prizes. This is not to say that these prizes, nominations and other signs of recognition are not deserved. But it is to warn against the Matthew effect: ‘the accruing of greater increments of recognition for particular scientific contributions to scientists of considerable repute and the withholding of such recognition from scientists who have not yet made their work.’ The Matthew effect derives its name from the first part of the following sentence from the Gospel according to Matthew (13:12): ‘For to the one who has, more will be given, and he will have an abundance, but from the one who has not, even what he has will be taken away.’

One consequence of this effect is that collective endeavours are attributed to the most famous of the co-authors or that when several individuals work at the same time on the same problem, the author who already has the biggest name receives the credit for the work which in fact has been undertaken by a collective. The Matthew effect reinforces the idea that academia is about individual geniuses, denying how ideas are almost always co-produced in scholarly communities.

The second part of the biblical sentence recognizes that the effect is not merely that some get more by virtue of having a lot, but also that, as a result, others get less than their due. The Matthew effect has been particularly harmful for minorities. The effect on women – neither included in the biblical reference nor in the original article on the Matthew effect, and still a minority in the higher echelons of academia – has received its own name, the Matilda effect. In the history of science, women, including Matilda Gage – the 19th-century American suffragist and feminist critic whose work inspired the name – have been notoriously underrecognized for their contributions to science. Studies on the gender citation gap suggest that this effect exists in academia beyond the hard sciences.

Interpreting CVs: Asking the ‘Why’ Question

Another danger of overreliance on CVs is the risk of false inferences. Some CVs show gaps: a fellowship, a visiting position, and then several years without a permanent position. There is a tendency to draw adverse inferences: there must have been reasons why the applicant has never obtained that permanent job, reasons that then also stand in the way in this round of hiring. However, without asking the applicant or referees the ‘why’ question (for instance, moving because of a partner’s career, illness, etc), there is a risk that the applicant is punished over and over again for not having obtained a permanent job at the expected stage in a career.

Changing the Culture

Members of appointment or selection committees will shudder at the idea of having to read the PhD theses or publications of, say, 500 applicants. Membership of such committees tends to be enormously time-consuming even if one reads only application letters and CVs. Let alone having to read everyone’s work in order to get to their ideas!

But there are other ways of getting to ideas. Ask applicants, for instance, for a narrative CV, in which they spell out their most significant scholarly contributions. Revealing, then, is what they choose to highlight: what does this applicant consider important? Does that match the appointing/hiring institution’s vision? For instance, an article’s number of citations, its contribution to the literature, its impact on society, or its scholarly reception? Or having mentored a struggling student who eventually passed their exams? Or having led demonstrations against limitations of freedom of speech on campus? To make it easier, hiring institutions can state explicitly that they commit to the DORA principles and are therefore not interested in certain quantitative indicators, for instance, journal impact factors. Hiring committees can also decide to give more attention to research proposals: what are this applicant’s ideas for the future?

 In processes where there is a shortlisting stage, the next phase can then involve all committee members reading some of the writings of each shortlisted candidate. If one is confronted with fellow panelists who are pushing candidates on the basis of numerical or Matthew-type arguments, ask them which of the applicants’ ideas have stuck with them. One might actually get interesting committee meetings! And if one is really impressed by an applicant’s ideas, but there seems to be an unexplained gap in the CV, ask the applicant or their referee the ‘why’ question. The results may be positively surprising – and make all the difference in one person’s career.  Shall we call it the Why effect?

Thanks to EJIL’s Managing and Associate Editors for helpful input.