A Facebook friend recently shared with me a news feature article from Nature regarding the lack of replication studies in many fields of science, especially psychology. The article is from last year but still relevant and can be found here: http://www.nature.com/news/replication-studies-bad-copy-1.10634
I can definitely vouch for the article because I have also found that replication studies in psychology and psychiatry, especially those that do not produce significant results (i.e., “null results”), are very hard to publish.
Let me explain with an example. Let’s say that, as a researcher, you notice a previous study has found that depressed people have higher resting heart rates than people who are not depressed.
Maybe you decide to try and replicate the study using a similar sample (let’s say they’re middle-aged adults of approximately equal numbers males and females), and similar methods (using the same ECG equipment, amount of time measured, same signal filtering techniques, and same interview to determine if people are clinically depressed or not). There are two scenarios:
1. If you get the same results as the previous study, that the depressed people had higher resting heart rates than non-depressed people, it will be very difficult to get a journal to publish your results. Why? They aren’t novel. The reviewers will likely say things like, “This has already been demonstrated in previous studies”, “The results are not novel”, and “I have a better idea for a completely different study, please explain why you didn’t do that”.
2. If you get a null result, that is that depressed people did not have statistically significantly different heart rates than non-depressed people, that will be equally as difficult to publish because null results just aren’t considered very exciting. I think this latter problem is an even bigger one because, as the Nature news feature pointed out, it makes it very difficult for a study that originally published false significant results to ever be discounted. This is already hard enough to do even when null results have been published in replication studies, and the original study has been shown to be fraudulent (and I want to be clear that replication studies that show different results from the original study don’t necessarily mean that the original study was done improperly or made up – sometimes you can get significant results that aren’t true simply by chance).
Case in point (or rather, case that has gotten entirely out of hand and has caused a lot of problems): An original study published in 1998 in a high profile journal, The Lancet, appeared to show that autism could be caused by a vaccine. Many replication studies done afterwards with rigourous methodologies and larger sample sizes showed no such result. So an investigation took place into the original study and its author, Andrew Wakefield, and they were surprised to find that he had done pretty much every single thing a scientist is never supposed to do, including manipulating data, not declaring conflicts of interest, and stealing other scientists’ lunches from the office fridge. I’m not entirely sure about the last one, but one of my favourite claims from the paper included suggesting a gastrointestinal disorder was a mechanism linking the vaccine to autism, despite the fact that the grand total of 12 subjects (massive recruitment effort there…) that made up the study were referred from a gastrointestinal clinic. The study has since been fully retracted and the author has lost his medical license. I assume he has to make his own lunch now.
So imagine if no journal had bothered to publish those replication studies, simply because they weren’t interesting. Maybe no one would have looked into the original study’s faults and we would have never realised it was a massive fraud and and still believed that vaccines cause autism. Luckily the issue was such a controversial one that the interest factor was high enough to publish the many null results in replication studies that came after. So even though a lot of people still believe vaccines cause autism, the general scientific consensus is that they do not. Sadly, just because scientists know that the false claim has been rectified doesn’t really undo the damage that has already been done in terms of public knowledge. Vaccine rates have dropped a lot since publication of the study, although some scientists in Australia are doing their best to change this.
But what about all of the many other topics and areas of research that are not so controversial? How much misinformation are we dealing with on a day-to-day basis simply because of this ridiculous precedent to ignore null results? Give them some love!