Diet research is a difficult thing. Researchers try to account for all the variations, but unlike a drug trial … where you can give one group a placebo and another the real thing … dietary research often relies on questionnaires, diaries and surveys. Some studies use a “meta” approach and compile the results of dozens of such studies, averaging the results between them.
Its an inexact way to do research in the first place, and the studies themselves usually point this out in careful language in the conclusions. Those carefully crafted sentences are rarely included in the popular news stories the media puts out.
One thing researchers have to keep in mind is the possibility that people will not remember correctly or cheat, as the Wall Street Journal’s Health Blog recently pointed out
The kids were supposed to be wearing pedometers to measure the number of steps they were taking each day. But some of those in the study got the bright idea to clip the pedometers to the collars of their pet dogs, upping the distance the youngsters appeared to be moving each day, according to the BBC.
Even worse, some studies are epidemiological studies (often called observational, prospective or cohort studies) where two existing populations are compared and tested for a particular thing. Dr. Mike Eades uses the example of two populations, one with high vitamin C in their blood and one with low vitamin C. The researchers track to see which group has the fewest colds, and publish the results. The study is bound to fail. As Dr. Eades says:
They try to think of all the differences between two large populations of subjects so that they can statistically negate them so that only the observation in question – the vitamin C level in the example above – is different between the groups. Problem is they can never possibly think of all the differences between the groups. As a consequence, they never have a perfect study with exactly the same number, sex, age, lifestyle, etc. on both sides with the only difference being the study parameter. And so they don’t really ever prove anything.
Most of these studies are false. In that their conclusion, as reported by the media, cannot be supported by the evidence presented in the study. That’s why you see conflicting reports on the advantage of vitamins and diet plans. (See this journal article I found linked on Dr. Eades blog for an explanation of why most research articles are false.)
We often think that science works by fortuitous discovery, where a lone scientist spills his martini and it mixes with a polymer on the desk and he discovers the cure for gout. He calls his buddy and everyone cheers, and all the scientists honor him with an award dinner where they all agree that Dr. Martini Spiller has the right idea.
The real way much of science works is that a study reaches a conclusion, and is published. The scientific community looks at the study, and some decide the author must have had a few martinis before the one he spilled, and his conclusion is attacked. The “attack” is often with another study showing a different conclusion. Both of these studies may be false, or at least unsupportable by the facts they present. After a while there is a winnowing out of claims and “truth” comes out. Its a messy, unromantic process that wouldn’t make a movie producer happy.
Often, its a bit more colorful. In Simon Singh’s wondeful book Big Bang: The Origin of the Universe
, Singh recounts how Albert Einstein remarked that Georges Lemaitre
, one of the early proponents of the Big Bang theory, was a “moron”. Later, Einstein embraced what became known as the Big Bang theory. (Interestingly, the term “Big Bang” was a slur intended to show how naive and wrong-headed Lemaitre’s “hypothesis of the primeval atom” was. “Big Bang” looked better in the newspapers, so the media used that term).
Conflicting studies are frustrating for those of us who want to understand the science behind what works; it doesn’t help that what is “true” is frequently updated as knowledge increases. But such is the way life works. Read research with a critical eye, especially if you are reading a recap or synopsis provided by others. Even in the original research report, pay attention to the details in the conclusion.