Evaluating Claims Of Health Journalists
A few weeks back I wrote an article detailing my beefs with the modern day health and medical research enterprise. My criticism is not so much that there are big problems in the world of academic medicine/health (though there are many big issues) but that - based on the known issues - it is not acceptable for health and medical experts to assert a high degree of certainty about their research findings to the public.
(Readers ought to note I’m discussing research that has to do with interventions to improve human health - and whether or not they work).
When it’s your job to detect what works and what doesn’t work (in terms of interventions), but you have very faulty detection instruments, it is right that the public take a skeptical stance to your claims. And, I would argue that you owe it to them to instruct them to be skeptical about your claims.
One of my readers (my brother, actually!) responded to my article, “Very well, but it would be much more useful to me if you showed me how to spot a poorly substantiated claim versus one that actually stacks up!” (paraphrased). This was to do with me whingeing about how health related media articles often misapply research results, making incorrect and sensationalised inferences from scientific studies.
So, not being one to shrink from a challenge, I’m writing this article about that topic.
How can you know whether media articles reporting on health and medical research are fairly and accurately representing the research results they are writing about?
This is a big, complicated topic. I’m just penning out some preliminary thoughts here. Here goes.
Correlation Vs Causation
If you read a newspaper article which makes a claim that “X causes Y” based only on observational research, you can almost certainly disregard that claim.
You cannot know from observational research that “blueberries reduce the risk of arthritis” (or something like that - “masks definitely work” hehe) because of confounding variables. That is, the sort of person who eats blueberries does other things you haven’t looked at which lower their arthritis risk. Plus, rich people eat blueberries. And being rich confers all sorts of benefits.
I’m merely trying to illustrate that it is very, very difficult to account for confounding factors in observational research. I think you ought to assume - as a general rule - that it’s technically impossible. And if you hear this statement - “our modelling shows….” - then probably best to block your ears and shout “LA LA LA!”.
I do, however, acknowledge my belief that sometimes correlation can be so strong (as in smoking causing lung cancer) that it may be responsible to assume a causative link based on observations alone.
Does The Media Claim Match The Research Claim?
An honest journalist will report on claims being made by the researcher, and not put forward any claims that ARE NOT made by the researcher.
In the age of click bait, medical research is often misrepresented by the media for the sake of a sensationalist headline. This occurs in spite of the researchers, not because of them.
Sufficiently Large, Comparable Samples
When assessing reporting on interventional research, you need to know whether the sample size was large enough in each arm of the study. If they were not, the results may be a chance finding.
This area of research is called power. In order to evaluate whether a study is adequately powered, you need to have a fairly high degree of statistical expertise, which the average reported (and even the typical doctor) does not have. (I certainly do not have it either).
This is worth pausing to emphasise as it’s actually quite shocking.
In order to truly understand the research, you need to understand statistics.
Very few people understand statistics well enough to understand the research.
This includes the vast majority of regular Physios, OTs, doctors, nutritionists, nurses etc etc etc.
I recently skimmed a paper evaluating an intervention in chronic low back pain. I have a BSc in Physio and some post grad education in statistics (although I have not practiced as a Physio since 2005). But, I reckon I’m sufficiently interested in, and well read on, statistical matters to be at least on par with the average circulating clinical Physio.
I read this in the statistical analysis section,
“The primary analysis used a heteroscedastic, partially nested repeated measure, three-level, linear mixed model to assess the effect of group allocation on activity limitation at the primary timepoint of 13 weeks and additionally at 3, 6, 16, 42 and 52 weeks.”
In order to understand that paper, you need to understand that sentence.
I’ll eat my hat if 1% of clinicians understand that sentence.
Randomisation
Journalists need to know whether a study was properly randomized before they can really evaluate any claims it makes. You need to know that they know how to do that. I don’t know that they know that…
Blinding
Journalists need to know whether the right people were blinded in the study (for example, the people receiving the placebo treatment should not be told, “you’re in the placebo group”). Don’t have blind faith in journalists who do not understand blinding.
Appropriate Statistical Analysis
Was the statistical method cherry picked to secure a desirable outcome? You’d need to know it wasn’t. We are meant to be able to trust the journals to take care of this. A reliable journalist would stick his or her nose into this matter when they’re reporting on study outcomes.
Valid And Reliable Outcome Measures
Is the phenomenon being studied with accurate instruments? Do they accurately measure the thing you’re studying? Will they give the same measurement across different people and across time (assuming the thing being measured does not change)?
It is not to be taken for granted that they will.
Trustworthy journalists will concern themselves with this sort of thing.
Are The Results Significant AND Meaningful?
Statistical significance does not mean the result is clinically meaningful. An intervention can work A TINY BIT.
If no attention is given to the size of the effect you can disregard the media article.
Bottom Line: I Don’t Think You Can Be Certain
The medical and health research system operates on the assumption that you can trust the scientific journals. You are meant to be able to take it on faith and trust that the journals will do a good job of verification. It is readily apparent that’s not the case.
Because of the substantial issues in health and medical research, I believe that (with the current system) a very high degree of certainty about research claims is simply not possible. In my view, these things would need to be addressed as a minimum:
There needs to be in-person auditing of data collection at the time of research. A truly independent third party needs to check that data collection is not falsified at any point in the research process. Yes, that means a person peering over the shoulder of the researcher as they punch the results into a computer or write them down on a data collection sheet.
Each study needs to register which statistical technique they will choose to analyse their results BEFORE the study takes place and be held to that promise.
All research data needs to be publicly available for re-analysis AFTER the research has occurred.
Similar to point 1, there would need to be independent checking about the characteristics of the sample groups. In other words, it needs to be verified (by people other than the researchers) that the characteristics of the samples (age, sex, socioeconomic info, height, weight etc etc) are accurate. This is to ensure an apples vs apples comparison across groups - intervention, placebo, control.
I note that all of these things would be really, really expensive and impractical. But that would be what it would take to talk about study findings with the degree of certainty we hear in the present.
In the meantime, the bottom line is that certainty about health and medical matters is in short supply. I suggest that you apply a grain of salt to everything you hear, and that you attend your ear to people who speak honestly about the degree of uncertainty, whether they be doctors or journalists.