Research Shows?

Over the years, I’ve read a number of popular science books relating to physical performance and health. This sort of book is often about translating the results of scientific research into a palatable form, and showing why and how it matters. It can be done well, like Run Like A Pro, or poorly like No Sweat. 

One of the reasons why No Sweat is not a great book is the way it puts forward research results as being completely true, unbiased and totally trustworthy - beyond legitimate question. 

It is common to find expressions such as these:

“Research shows that it’s better to exercise every day than every second day.”

Or,

“Science says that you should get your heart rate up very high at least once per month.”

So, what’s the problem here?

This sort of reasoning is an overtly sanctimonious form of communication. But that’s not the MAIN issue. Even if the sanctimony was dialled down a little, it is still fatally flawed.

The fatal flaw is that this way of speaking belies a belief that “science” (which is usually short hand for research undertaken by universities or similar institutions) is unbiased and objective, yielding results and outputs that are highly trustworthy.

The fact of the matter is that the process of scientific research into human health, medicine and human physical performance, undertaken by universities and other research institutions is very often quite biased. Hence the sanctimony. Science is a process that can (but does not always) fumble itself to the truth. Importantly, when it comes to human health and performance, the truth about a particular subject can take many years to wash out. Take the sugar vs fat war as an example.

This matters because the level of uncertainty about research findings is often a lot higher than the authors of papers convey, or the reader takes away. The degree to which a finding is uncertain is far, far higher than is often conveyed by health and medicine journalists, who have a bad habit of running stories on health matters that read like statements of fact and not areas of conjecture, uncertainty and provisional knowledge.

The public can very easily be led to believe that research results are rock solid, when in fact they can be tenuous. If the general public incorrectly (and understandably) assumes that anything put forward in a scientific journal (or news reporting on such journals) is gospel truth, then they’ll be repeatedly misinformed.

Why am I writing about this? 

I’m motivated to pen my thoughts on this topic because I think this sort of misrepresentation about high levels of certainty in scientific research reflects the high degree of hubris in these institutions.

These people are behaving like naked emperors, and you ought to know it.

Personally, I find this level of hubris pretty galling. My personal experience in the Physiotherapy profession is that commercial interests often lead people into modes of thought about clinical matters which dovetail into their personal profit. You can make more money off people when you project a high degree of (unfounded) confidence in your diagnosis and treatment.

If humility is a virtue, scientists (and communicators of scientific findings) can become more virtuous - and more worthy of trust - by being honest about uncertainty and the extent of bias in the enterprise of modern day research. Additionally, people who are keen to point out the problems in trusting tradition, clinical experience and expert opinion should be up front about the biases which beset “objective research” as it is done in real life, as opposed to how one can conceive of it in theory.

I’ve stated that scientific research into health and human performance is often very biased. In the rest of this article, I’ll explain why by giving a few reasons for why this is the case. It is not an exhaustive list, just the bare minimum of the issues I’m aware of.

Publication Bias

Medical (or allied health, psychological etc) journals are strongly biased against publishing negative findings - studies which found that something DIDN’T work. They like to publish studies showing that the intervention being tested DID work. This means there are lots of “negative” studies sitting in the top drawers of desks all over the world which have not seen the light of day in the journals. And it matters because we need to know when a study finds an intervention DID NOT work. Such a finding could help us to temper our enthusiasm for an intervention that has many positive studies in the published literature. Such a study could help to prevent wasteful allocation of resources to a failed intervention. Such a study - were it to be published - could help to prevent some poor hapless pHD student from repeating the same study! When “negative” results are severely misrepresented in the scientific literature, the waters become muddied and the likelihood that the truth is known about something decreases.

The Bar For Statistical Significance Is Too Low

Firstly, a word about context. Statistical significance is a phrase that’s associated with studies which evaluate the effectiveness of interventions. This is often done in comparison to a control group and a placebo group, so that outcomes across intervention, placebo and control groups can be compared.

What is statistical significance, then? 

It is;

• a measure of probability, from 0 to 1;

• which expresses the likelihood that an observable difference between two groups (e.g. a treatment arm of a study and a control arm of a study) is NOT DUE TO THE INTERVENTION, but to chance.

In order to be confident that the seemingly positive effect of the intervention you’re studying is not simply a product of random chance, you need to be certain that the probability (likelihood) of this is low.

The established norm for statistical significance in medical scientific research is 0.05. In other words, if it can be determined that there is a difference between groups in a study (e.g. intervention vs control), and that the probability of this difference being explained by randomness is less than 5%, then result is declared to be statistically significant - “we are 95% sure this isn’t just a chance finding”.

In recent years, people such as John Ioannidis have criticised this norm. They state that there needs to be a higher bar for interventions to clear in order to be deemed statistically significant. They make the claim that, if the bar were to be raised and findings of the past re-evaluated, a great deal of “positive” studies would fail to maintain that status. Things we thought worked do not in fact work.

You can find out lots more about that here.

Publish or Perish Incentives

You may be familiar with the expression, “publish or perish”. Here’s what it means. People who are employed by universities to do research need to publish journal articles. They need to do this because that is one of the principal means by which the performance of individual academics and universities as a whole is evaluated by people and governments who bring income into these universities. 

The way you become “someone” in research environments is to become a published author. And, as one of my lecturers once told my Physiotherapy year group, “if you’re not someone, you’re no-one.”

There are very negative consequences which flow from this fetish for publication. 

A lot of unimportant research is done simply for the sake of going through the motions and bolstering one’s CV. The pressure for academics to continually be publishing is one of the factors that have led to the advent of predatory journals - which are keen to charge academics for publishing their work, but provide very poor quality (if any) professional publishing services (such as peer review). It is worth highlighting that there are legitimate journals which accept payment from authors/academics to publish their work (providing it can pass their peer review process). The incentives of such an arrangement are quite dubious, in my opinion. 

Another negative consequence of the obsession with publication is that lots of very unimportant studies get funded and produced. There is a huge amount of low importance research being done, which is quite an extraordinary waste of money. It is a shame that the modern university has such a low view of teaching, preferring to spend the time of academic staff on time-wasting research over teaching students.

More importantly - for the purpose of this article - an obsession with publication, combined with publication bias, means that “negative” results will be subjected to data massage to make them positive. They won’t get published otherwise, which hurts the academic and the university. Either that, or statistical analysis techniques which favour a “positive” result will be favoured over others. Because “negative” studies will not be published, some interventions (maybe many) which do not work are misrepresented as being effective so that the researchers can get published. The net effect of this is that consumers of health resources and information (punters like you and I) are misled.

You can read more about that here.

Replication Crisis

Many landmark findings in the world of healthcare have not been replicated, despite the fact that replication has been attempted. If the original findings cannot be replicated, doubt is cast over the original results. It becomes more probable than not that the landmark finding was … not so landmark. That the silver bullet treatment was effective, yet only for a few people and not nearly as powerful as was originally thought. And that it was not such a breakthrough after all. But the train has bolted: healthcare professionals and the public have already formed an undeservedly rosy picture of the intervention.

Anyway, I hope you’ve enjoyed reading my thoughts. I have more to say but no time to say it! I’m sure I’ll be thinking and writing more on this topic in years to come.