Weighing Up The Quality Of Evidence
The articles below were cited as supporting evidence by a vegan during a twitter debate with me last week. So I took a look at the first six, in order to see if the support they provided for his position was strong or weak. What I found was quite revealing.
These are the articles I looked at, they are the top 6 on the site he referred me to. Below these follow a series of others taking up particular topics such as auto-immune diseases, cancer, diabetes and so on.
I can't remember or find the details of the debate, but it included the claim a plant-based diet is superior to one centred on meat, and that there are a stack of scientific studies to back this up. He then referred me to this webpage.
So let's look at these six articles and see what kind of support they provide for his argument.
1. A very-low-fat vegan diet increases intake of protective dietary factors and decreases intake of pathogenic dietary factors. (available here)
This opens with,
"There is increasing evidence that dietary factors in plant-based diets are important in the prevention of chronic disease."
Sounds promising. What is this 'increasing evidence' ?
"This study examined protective (eg, antioxidant vitamins, carotenoids, and fiber) and pathogenic (eg, saturated fatty acids and cholesterol) dietary factors in a very-low-fat vegan diet."
The problem here should be obvious. The protective and pathogenic factors indicated are precisely what is in dispute between proponents of the two diets. In other words, the study presumes what it has to prove. Supporters of an animal-sourced diet do not accept the diet-heart hypothesis, that saturated fat and cholesterol cause cardio-vascular disease. The reasoning is entirely circular.
The circularity is evident in the study's methodology and conclusion. What the researchers did was feed the subjects a diet designed to deliver the desired result, which it did.
"These results suggest that a very-low-fat vegan diet can be useful in increasing intake of protective nutrients and phytochemicals and minimizing intake of dietary factors implicated in several chronic diseases."
This weakness applies equally to countless studies that make up what mainstream nutritionists call the 'mountain of evidence' in support of their position. On close examination it also presumes what it has to prove, it rests on assumptions that are highly contested.
What I find interesting is that this paper is listed a the very top of the webpage, which means the site's owners must have had a high level of confidence in its value as supporting evidence for a plant-based diet. On examination, however, the study is worthless, of no use whatsoever. How did they miss that ? This is 'Arguing 101', circular reasoning is the most basic of errors.
I will return to this theme, for the moment lets continue.
2. Position of the Academy of Nutrition and Dietetics: Vegetarian Diets (available here)
This document turns out not to be a study at all, but a position paper adopted by the Academy of Nutrition and Dietetics. Its inclusion here is nothing other than an appeal to authority. This only works, however, if the body in question holds some authority in the eyes of those you are trying to convince. It does not. This Academy is a pivotal institution within the nutrition mainstream and to critics such as myself and others is totally discredited.
An appeal to authority is a very weak argument at the best of times. In this case it is a very poor one, because it only has force among those who already accept it. It is preaching to the converted rather than seeking to win over new converts.
3. Meat consumption and mortality–results from the European Prospective Investigation into Cancer and Nutrition. (available here)
This was a large scale European study that looked for an association between meat, processed meat, poultry, and all-cause mortality. The value of epidemiological studies of this kind is not considered high, at best they can be used to generate hypotheses to be tested using more rigorous trials, at worst they are completely misleading. Nevertheless, this one we will take at face value. Here is its finding,
"After correction for measurement error, higher all-cause mortality remained significant only for processed meat (HR = 1.18"
1.18 is a very small association, for meat the number was 1.14 and this was discounted as not statistically significant. There was NO association found for poultry. In other words, the study supports a diet of red meat and poultry, it is evidence in favour of a meat-centred diet, through its failure to find a negative correlation. This obviously does not apply to processed meats, but a carnivore position does not entail a commitment to processed meats, it is hostile to processed food in general so this finding can be accommodated without any difficulty.
As with study No1, we seem to have here an inability to understand what makes a good argument.
4. Association of Animal and Plant Protein Intake With All-Cause and Cause-Specific Mortality (available here)
This was another large study, of US health care professionals, and looked for an association between protein source and mortality. The conclusions of this study appear to offer much stronger support for a plant-based diet than we have seen so far,
"High animal protein intake was positively associated with cardiovascular mortality and high plant protein intake was inversely associated with all-cause and cardiovascular mortality,... Substitution of plant protein for animal protein, especially that from processed red meat, was associated with lower mortality, suggesting the importance of protein source."
This result, however, is undermined by the following statement, also contained in the paper's abstract, "After adjusting for major lifestyle and dietary risk factors, animal protein intake was not associated with all-cause mortality."
So, animal protein might increase your risk of dying of a heart attack, but overall your life expectancy is unaffected. This must also mean that animal protein is beneficial in some ways, to balance the numbers out.
Furthermore, the actual figure given for cardio-vascular risk is 1.08, smaller than the one discounted as insignificant in the European study. But worse is to come, for the study also states this,
"These associations were confined to participants with at least 1 unhealthy lifestyle factor based on smoking, heavy alcohol intake, overweight or obesity, and physical inactivity, but not evident among those without any of these risk factors."
In the trade, this is known as a 'confounder', it completely invalidates the conclusions of the trial, according to the normal rules of science. The study has NO VALUE whatsoever, any association it found could be due to any one of the lifestyle factors and not the protein source, it tells us NOTHING, about anything.
5. Nutritional update for physicians: plant-based diets. (available here)
As with No 2, this is not a study either, and its inclusion on the list is even more odd. It consists of an article urging doctors to advise patients of the benefits of a plant-based diet. As it explains,
"Despite the strong body of evidence favoring plant-based diets, including studies showing a willingness of the general public to embrace them, many physicians are not stressing the importance of plant-based diets as a first-line treatment for chronic illnesses. This could be because of a lack of awareness of these diets or a lack of patient education resources."
The update seeks to remedy this, and includes a case study followed by a literature review. This draws on much of the same research used by the Academy Position Paper above. The publication it appears in seems to be owned by the health provider Kaiser Permanente, which may explain this passage,
"Concerns about the rising cost of health care are being voiced nationwide, even as unhealthy lifestyles are contributing to the spread of obesity, diabetes, and cardiovascular disease. For these reasons, physicians looking for cost-effective interventions to improve health outcomes are becoming more involved in helping their patients adopt healthier lifestyles."
Whether physicians found the update convincing or not I have no idea. Why the owners of 'Plant Based Health Australia' would think this would be persuasive to neutral readers in a discussion over the optimal human diet I am at a complete loss to explain. The article simply assumes precisely what it has to demonstrate, once again it is preaching to the converted and only the converted. This can be seen in the following sentence, "The goal of this article is to review the evidence supporting plant-based diets and to provide a guideline for presenting them to patients." The nature of this 'review' should be obvious - it is not a critical inquiry into the evidence and its merits, the outcome has been pre-determined. Instead it seeks to overpower any hesitation physicians might have through no less than 46 documents cited, all confirming the one point of view. No objections or counter-positions are even considered in this 'review'.
6. Association between dietary fiber and lower risk of all-cause mortality: a meta-analysis of cohort studies. (available here)
This time we have a meta-analysis, which are generally considered as a good form of evidence although they have their limitations too. The trouble with this study is of a different kind - relevance. It is irrelevant to the matter at hand, or in fact any matter at hand. It is another worthless study.
The trial looks at fibre and seeks an association between the amounts of fibre in the diet and all-cause mortality. It does find a relationship of some kind but this does not help. The reason for this is that it compares fibre with... fibre. This means that all it is able to show is that some amounts of fibre might be more or less beneficial than other amounts. What it does NOT show, one way or the other, is whether fibre has any benefits when it is compared to other kinds of foods. Since our discussion is over the merits of plant-based diets, as opposed to animal-sourced, it tells us nothing whatsoever.
There is a consistent theme here, throughout all six papers. NONE of them serve the argument they are meant to be supporting. This is for a range of reasons in each example, but it applies in every single case. These documents have not been presented out of general interest, or for information purposes, or for entertainment, they have been selected specifically as 'key articles supporting plant-based nutrition'. They were chosen by advocates for this diet, not by me, not by an opponent, to support the argument for their way of eating, and they do nothing of the kind.
There are arguments in favour of a plant-based diet, there is evidence that supports these, it does exist, but it is not to be found here. I find that interesting.
In another post, (see here) I took the authors of my mainstream nutrition textbook to task for their poor arguing skills. Here I am doing the same again, this time with supporters of a plant-based diet. I think part of the problem here is that neither of these groups of people, those who've made careers in the nutrition establishment, or those in the vegan community, get much practice in arguing. As a result, they have little idea as to what makes a strong argument, and what makes a weak one.
I think the Physicians Update is the document that captures the issue best. Its weakness consists of a complete failure to engage with the other side of the debate. It looks only at supporting evidence, this is typical of the mainstream too. It means they have no real idea of their opponents' thinking, or the weak points in their own argument, or for that matter their strong points. I honestly believe there are vegans who genuinely think the health terrain is their strength, whereas I would say, as an adversary, they are on much stronger ground when they talk about animals, that's what wins them supporters. The diet side is not an asset, its a liability, but they lack this self-awareness.
In this respect our side has the advantage. That is because we are outliers, we are going against the stream, which means that to get where we are now we have had to use our critical faculties, we have had to challenge dogmas and orthodoxies, we have had to think and rethink our own positions. Many of us have travelled a journey with several waypoints along the way, we have dropped ideas that turned out to be mistaken or misguided, we have developed a certain humility, we know we can be wrong, we often are, but we also know we are onto something, we are not afraid of digging for the truth, wherever it leads.
We've also learnt something about the skill of argument, because we've had to defend our positions repeatedly from hostile attacks, from ridicule, from authority, and do so with words, with ideas, with evidence that really is supporting evidence. If we've tripped up, we have been punished for it.
In case you haven't picked up on it yet, this blog is one long set of arguments, one after another and another. I picture myself as in the ring (I do train in boxing) throwing out jabs, looking for an opening for a big right, dodging and weaving. Some of my jabs miss, some are poorly timed, some are misdirected at the wrong target, no matter, I am refining my art, post by post. Speed, precision, power, balance, timing, I am working on them all every day.
We will win this fight through superior arguments. Science is important, sure, but only to the extent that it provides evidence for arguments, and is harnessed correctly to appropriate arguments. The examples I have provided above are of how NOT to do this. Please learn from them, do not repeat such errors, master the art of argument.