The ins and outs of peer review
Throughout the whole saga of the UEA email debacle I noticed how little understanding there seemed to be in the media or the general public of how peer review works and what it can achieve. In that particular case a good understanding of peer review could have led most people to realise that there was at least a good possibility that nothing dodgy was going on at all at UEA, as I have written before. But even within science there is debate about peer review, how it works, whether it works, if it should be changed. So I thought that as an ex-editor of a peer-reviewed journal it was about time I put my own personal views forth, as well as hoping to educate those less familiar with the topic.
Firstly I had better define what is meant by a peer in this context. Academics are usually the ones subjected to peer review so peers are other academics or occasionally industrialists researching related or similar subjects. This fact in itself brings up one of the criticisms of the peer review system – namely, these people are often in competition so how fair is it allowing them to judge the quality of their competitors’ work?
Peer review is used not only in journal publications but also to decide upon the award of financial grants e.g. by the government funding agencies such as the EPSRC. I’m going to use examples from journal publishing but you can see how the principles apply equally to both.
In brief, here’s how it (should) work: Researcher does research, writes it up accurately and submits it to a journal for peer review. Journal editor sends paper to two or more peers, who provide their opinions (reports) on how good/relevant/useful the research is and on how it can be improved. Journal editor then makes a decision based on these opinions about the suitability of the work for their particular journal, and passes the comments of the peers (referees) along with his/her decision to accept/reject/ask for revisions on to the author.
Depending on the journal and the area of science you are working in, the qualities being looked for will change as will the ability to appeal any decision made, but broadly speaking any editor will look for the work to be a) original i.e. not done before and b) ethically sound (not stolen, plagiarised, harming animals unecessarily, etc.), as well as c) in keeping with the expectations of the readers of the journal in terms of what kind of research they expect to read there. Of course some would claim that there is also d) will this enhance my journal citation record? to consider, though I would argue that this falls into c) as readers will cite work that they find appealing and relevant to them.
All things being fair and equal, the peer reviewers should provide an unbiased report that explains what is new, why it is important, how this leads on from previous work etc. The journal editor, not being an expert in everything, is more or less dependent on the reviewer comments. Some editors are researchers in their own rights, which means they have expertise in certain fields but can never have it in all fields. Editors develop a feeling for how much they can trust certain individuals and how much emphasis to put on any single report. Editors have the final say, so they must also be incorruptible. This may sound like a hard quality to find, but the point is that an editor who is partisan or seen to be partisan will lose the respect of their community and no-one will want to publish in their journal. Being seen to be partisan will therefore destroy the journal as well as the editor’s reputation. Therefore it is on the impartiality of the editors that the fairness or otherwise of the peer review system relies.
It also relies on the reviewers first and foremost to give fair and impartial advice. To this end often reviewers are promised anonymity so that their names are not revealed to the paper authors. This should mean that they can say exactly what they think without fear of recriminations. I have spoken with many many authors who believe that they are able to identify their reviewers for a particular paper. 9 times out of 1o they are incorrect. And I have lost count of the number of times I have been asked “Why didn’t you use the referees I suggested, they know about this area and the referees you picked clearly don’t” (always for a paper I have rejected on referee advice). At least half of the time, I HAD used the suggested referees. This shows you that anonymity does, at least to an extent, offer the protection and fairness it promises.
Misconduct isn’t always detected by this method – as evidenced by high profile, systemic cases of this such as Marc Hauser, Woo Suk Hwang, Jan-Hendrik Schoen, etc. But it sometimes is – these are the many cases that the public and even the scientists do not see.
However there are flaws to the system – no-one thinks it is perfect. Peer review is simply (for most people) the lesser of the known evils. And no-one has really come up with a satisfactory method that is better. And so it persists. There are variants on what I have described, the so-called single-blind peer review (single because only one side of the equation is anonymous – the reviewers know who the author is and this could affect their judgement). Firstly there is double-blind peer review. I don’t actually know of anyone who uses this as a large-scale, functioning system. I am told that it is virtually impossible to achieve, as most authors will give themselves away, if not by their writing style then by their chosen references (most people will tend to self-reference). Even if this could be avoided, because research is often presented at conferences before it is published, identification of the authors would still be possible.
Another system was trialled by Nature (their findings can be read here); that of open peer review. Here peers are asked to give their opinions without anonymity. While this is a system used by several medical journals including the renowned BMJ, Nature’s trial did not meet with an overwhelming welcome and the process was abandoned. It seems that scientific researchers were more comfortable with the devil they knew, as Nature found that from many disciplines, papers were not even submitted to the trial.
So why do people dislike the system so much? In my opinion, a large part of their unhappiness arises simply because an anonymous reviewer has “rejected” their work. The only system that could improve on this would be to publish everything in the place to which it is originally submitted. And hopefully most of you can see the problem with that idea. But there are other, more valid reasons to dislike peer review as it stands. Decisions are based usually on the opinions of two or three selected people. If a subject is controversial (as in climate change!) then the outcome of the review process will depend to a large extent on the choice of the referees, and thus on the editor’s personal opinion about the work (remembering that editors are suuposed to be impartial and may not be experts) and their familiarity with the field. If I get a controversial sounding paper I can easily facilitate its publication by selecting reviewers who I know to be in agreement with the views expressed. Or I can prevent it being published by selecting reviewers who I know will disagree with it. Or, if I am not familiar with the field, I might do either of these simply by accident, and not realise what I am responsible for. None of these options are really ideal. For me, the fairest solution here would be to send the paper to pro and counter referees and ask them to assess the paper as fairly as they can, then base my decision on their advice, the way they express it, what I know of those people, and necessary further consultations with non-experts who have an interest in the case e.g. a board member for the journal. If the decision is still not clear, an argument (and this is one that I have personally made) would be to publish the paper and let the community decide what to make of it. This might mean a few papers are published that later prove to be wrong, but until it can be clearly decided if this is the case, the benefit of doubt might be the fairest option. Such a course of action assumes that the editor is familiar with the community involved, and does not have any strong beliefs about the matter his/herself (or is able to suppress these). And is primarily concerned with fair treatment rather than with citations. (Publication of a controversial paper might result in increased citations because later authors cite it to agree/disagree with it, but then again it might not.) All of these are qualities that are probably quite rare in editors these days, but in my opinion they are the most important ones to look for when choosing your journal.
I also had a longstanding discussion with one particular author who claimed that he would not publish with a certain publisher because they did not treat his work fairly. Instead he preferred another publisher who apparently did treat him fairly. Now think about this carefully, it goes back to my initial comment about why people dislike peer review. I happened to have knowledge of both publishers and the treatment that author received at both of them, and I can state categorically that the publisher that actually treated him more fairly was in fact the one he would not publish with, because they dared to reject his work when advice said that it was not up to scratch; in other words, they treated him just like everyone elseand took each paper on its relative merits. The other publisher presumably took the view that keeping him happy was more important than publishing a couple of dud papers, as he would produce many more that were not duds and would prefer to publish there in future. They therefore gave him preferential treatment that a young researcher starting out would not receive, simply because of who he was. Is this fair? No, but it is what happens in life so perhaps it is not surprising that publishing should be any different; we are all judged by our past mistakes and triumphs. I would expect an employer to prefer to employ a candidate with a lot of positive experience on their CV rather than a complete beginner or someone with only a history of failures. And yet peer review, in it’s finest and fairest form, asks us to take each paper on its merits and without considering the history or future of the author. Should we be using a system that does not reflect true life with which to make decisions that will affect lives?
Ultimately peer review revolves around the editor and/or publisher of the work and it is these people/places that should be under scrutiny if the system is to be improved, as well as the authors and reviewers.
I expect that anyone who has managed to get to the end of this little lot will have something to say about it. I hope you will comment, and I welcome the discussion.Explore posts in the same categories: Uncategorized