Posted tagged ‘publishing’

Results null but not void

August 31, 2010

There’s been a bit of discussion recently in various forums, perhaps most visibly on Ben Goldacre’s Bad Science blog, about the usefulness or otherwise of null results. I thought I would add my twopennorth to the discussion.

So first of all what is a null result? Mostly when people use this term they are referring to a result that does not tell them anything that they want to know, perhaps it is even a negative result, though I would argue that these two are quite distinct, and that a null result is simply one that does not further the research one way or the other.

In medicine and pharmaceutical trials the null or negative result is clearly of great importance and you can see why companies producing the drugs would be motivated to hide or cover up such results – why would you want to tell the world if the very expensive drug you are hoping to sell to cure cancer is less effective than current treatments? That’s basically saying “don’t buy this drug” and if you are a business that’s not a good advert for your products.

This argument can be extended to anyone with a commercial/financial interest in the outcome of the research (I’ve touched on this in my review of Fiona Godlee’s Sense about Science talk). A way round it might be to make the testers independent of the developers. This is a system that might work in really big industries like pharma, but would not really be feasible for smaller, or niche industries.

I’m thinking of materials development, as a subject close to my heart. I think it is fair to say that if you are an aerospace company buying materials to make your plane with, you are not going to take very seriously the claims of the makers  of any materials you buy in, and will want to do your own tests. A null result means you won’t buy the product, end of story. If you are the company that makes the materials, and your own tests show the material to be worse than previous ones, you will put that project to one side and start again. Time wasted, yes, but probably things learned along the way. Most companies doing research build this in to their research budget. Even pharma companies recognise that only about one in several thousand lead compounds will ever make it to market.

But a world where null and negative results are still of great debate is in academia. As an academic researcher, you get a few grants per year and are expected to produce several publications in reputable journals so that you can continue to receive such grants in the future. The publications show that the grant award had been worthwhile. What is wrong with this?

Well for a start, if your idea doesn’t work as well as you thought it would, you have a problem. You might get a null or or even negative result. What do you do then? All that work and you can’t get a decent publication out of it, which means you can’t get another grant later on. At the very least, you would be encouraged to gloss over the results, carry on to the next thing, change tack. This happened to me during my own PhD. If you were less honest, you might become embroiled in some kind of data fabrication saga, claiming things that were untrue. How else can you salvage anything from those years of work?

The key point here is the phrase “decent publication in reputable journal”, which you need to sustain and continue grants. Most researchers and research agencies judge this by impact factor of the journal, and/or citations to the paper itself. Within this framework there is no room for null or negative results. Why would you spend the time and effort to write up an experiment that did not work out how you hoped, and does not really further understanding? Apart from, of course, to prevent others following the same doomed course of action. This may earn you some brownie points with peers, but no money. So most researchers don’t bother to write up null results. Even if they do bother, publishing them is not straightforward.

In the main journal I used to be Editor of, we used to look for novelty or advance in chemistry or properties of materials. These criteria were used to rule out an awful lot of “low impact” work that was basically “just another” synthesis of a nanomaterial or the like. We knew that these papers would not receive large numbers of citations and so we did not want to publish them in our journal. Most editors would be the same. Null results were obviously not wanted. Negative results would occasionally make it in if they were unexpected or in disagreement with previous work – then there is more chance that people will cite these works.

So it is the journals and their editors who are to blame? Well you could be understood for thinking that. But look deeper. Why do the journals want to publish high impact papers, and exclude those that they believe will not be? Simple really. If my journal does not have a good impact factor and a reasonable amount of content, who will buy it? University and industrial library budgets are being cut all the time and  librarians are looking actively for “unnecessary” journals. How do you decide what is unnecessary? Well normally this is considered to be low impact work, small niche journals, articles that will not be read or cited by their peers. Hmm.

So librarians are to blame? Not really, they are only applying the standards laid out by the academics themselves. The standards imposed by the funding agencies and their ongoing need to be able to measure research outcomes in numerical terms. A null result has a null numerical measure. Null benefit to the world, though? I think not.

Open all areas – thoughts on OA publishing

August 4, 2010

I’ve had quite a few conversations about open access publishing recently, which is odd because I would have thought that I would have this conversation less now that I am not an editor. Funny how things work out.

Perhaps the reason why this topic has come up more recently is because I was working in chemistry publishing, which apparently has the lowest take-up rate of open access (OA) of any discipline (sorry, not sure where I read this so can’t provide any verification of it – anyone who can that would be welcome – my experience is certainly that it is less popular in chemistry than in other related sciences). Now I’m trying to take a broader overview of science in general it becomes more of an issue.

  So what is OA publishing? Basically it means that an article, when published, is free for everyone and anyone who has internet access to read and maybe download. There are various models for doing this. One is that advertisers pay to appear on the site, and no-one else has to pay to publish anything on there. All OA sites can be peer-reviewed or non-peer-reviewed; for ease of discussion I am going to stick to peer-reviewed sites as anything else is effectively not much more credible than, frankly, a blog (yes that includes mine), i.e. it is unsubstantiated, unchecked opinion. You can discuss this if you like, but for me that’s another blog post, another time. Maybe I’ll get around to it soon, if anyone is interested.

An extension of this type of model is the one whereby a government funds the publisher directly so that no-one using it has to pay. PLoS is an example of this model. Note that a source of money is still needed or the publisher would not be able to operate the site.

Another model of OA is that those who wish to publish pay for the paper to be made accessible to everyone (as opposed to just subscribers) when the article is accepted for publication. Many reputable publishers, including those in chemistry such as RSC and ACS, use this model alongside the subscription-only model in a so-called hybrid system; that is, authors are offered the choice to pay only at acceptance of their article. If they don’t choose to pay then the article is accessible to subscribers of that journal only. The cost of funding this would then come out of an author’s research grant, rather than out of library funds, which is where traditional subscriptions come from usually.

Depending on your exact definition of OA, you could also include things like pre-print servers within open access; these are repositories for articles and discussions about them before the article is published, but the article often remains on the pre-print server in some form even after publication. A popular pre-print server I have come across a lot in my work is arXiv, which is run by Cornell University Library for specific communities such as physics, maths, etc. I’m not sure how this is funded but presumably it doesn’t require so much funding as it is so author-driven and no value (formatting, peer review, enhanced HTML content, etc) is added to the content before it appears.

So far so good then, hopefully you understand how the different models can work. And that if value is to be added, a source of finance is required. Even if no value is added, most researchers want to have their work, once published, held on a secure server that they know will be there “forever”. Even those who use pre-print servers generally still strive to get their work published eventually. And then there is the question of the mark of quality provided by peer review and publication in a respected journal.

There is a movement out there that says that as research is funded by taxpayers, then taxpayers aka the general public should have access to it. Again I think this is part of a larger discussion about who should have access to data – everyone wants to or just those qualified to understand it and use it correctly? But I can certainly see the argument for greater availability of information. Strictly speaking, of course, this means that only research funded by the UK government would be available to UK residents. They would still have to pay to access US-funded research. Not quite what this group are after, probably, as the majority of the world’s research is carried out outside the UK, though arguably the UK may lead the field in some areas.

Then there are some companies and organisations that are quite keen on OA. The Wellcome Trust is a notable example of this. They actually fund research as well, and they do make extra funds available to researchers to allow them to publish their work on an OA basis. But what about those more commercially oriented companies? They may or may not fund some academic research. They certainly would not allow any of their company secrets to be published in the open literature, pay-to-access or not. But it would be of great benefit for them if other people’s research in fields linked to their own were made freely accessible – they would save a small fortune in journal subscriptions, and would not have to pay any publication fees as they do not publish their own research anyway. Perhaps you can see why they would be keen on OA?

My opinion is that if funders want work to be made publicly available then they need to provide the money to allow this to happen, directly to the researchers who can tie the money to the work in question. Otherwise they can’t really object when charges are made for access to the information.

Partly this can be achieved through government research funders allocating a publication budget to sit alongside the research one. This should not be given directly to any single publisher (this will create a monopoly whereby people funded by that government can only publish with a certain publisher or small group of publishers) but should be made available to the researchers or universities themselves.

Another approach to combat the exploitation of OA information by big corporations would be for a government to impose some sort of research tax on any company within its jurisdiction admitting to performing research and development (R&D). This money could then be used to pay the publishers and authors whose work the companies are using. This would not be a popular choice, and I doubt it is one that most governments would hurry to introduce. But where else is the money going to come from?