Today I’d like to highlight a post from a few weeks ago on Neurobonkers discussing a recent meta-analysis suggesting that most neuroscience studies are underpowered, meaning that they don’t have enough subjects to make definitive conclusions about their results.
This really is a big issue in many fields, especially neuroscience and psychology. I wanted to chip in on this issue because it’s a problem in my field. I can’t tell you the number of papers (and more likely, posters) I’ve seen that claim seemingly interesting results based on…. 2 people. It’s shockingly poor science (and irresponsible, in a way) to try to suggest something based on a very small number of people. The problem with underpowered studies is that they have a higher chance of having what is called a “false positive“. Think of it this way, if you study two people and one of those people happens to be some sort of freak, then you can’t really take their average of whatever you’re measuring and generalise it to the rest of the population. The more people you study, the less likely this is to happen (although there is still always a statistical chance this will happen – it just gets smaller with larger power).
On the other hand, I feel like reviewers have an easy out when they want to reject a paper by saying that the study is underpowered when they have no evidence for this. Saying something is “underpowered” is only true if the number of subjects you have is not enough to see the effect size that you expect. For example, if the effect size of whatever you are looking at, say the difference in inflammation between depressed and non-depressed persons, is very large (i.e., the difference in levels inflammation between the groups is A LOT), then you need less people to reliably see this effect. If the effect size is expected to be small, then you need more people. So be wary if someone tells you that a study is underpowered without explaining why.
However, in the case of the paper mentioned above, the concern of the authors is that we may be misrepresenting results or even missing out on differences that are expected to be very subtle (and small) in the field of neuroscience, which I do think is a fair call.
When will neuroscience research become cheaper? Or when will governments begin to fund more research into this area?
More stuff about this:
- Serious power failure threatens the entire field of neuroscience (bps-research-digest.blogspot.com)
- Many Neuroscience Studies May Be Based on Bad Statistics (wired.com)
- Poor Statistics Undermine The Reliability Of Neuroscience (wmbriggs.com)