Explanation of the differences between a Prospective Trial and Meta-Analysis

This poorly designed study is being widely criticized by the scientific community.

Compare the following two scientific methods:

The National Eye Institute’s Age Related Eye Disease Study (AREDS) was a prospective, double-blind study. This means that it began with a random group of people, and that both the doctors and patients did not know whether they were receiving medication or placebo. Prospective studies are based upon what is known as the Null Hypothesis. What this means is that the study hypothesizes that no differences among different treatment groups will occur. When differences are in fact found, as were found in AREDS, and those results are statistically meaningful, a powerful tool for predicting what will happen if someone takes AREDS supplements is proven. No increase in the rate of disease or death was found in the thousands of people in the AREDS study.

This month’s article in Annals of Internal Medicine, in contrast, is a meta-analysis. It begins without making a Null Hypothesis. That is, it is not looking to prove any particular fact, but rather looks at results AFTER they have occurred and attempts to show a pattern. Furthermore, rather than being an analysis of one population, its results were even further weakened by analyzing results from many different studies performed on disparate populations, with different timelines and testing conditions.

Scientifically, the Annals study proves nothing. Of course, it makes for good reading in the lay press and newspapers.

Here is another way of looking at it:

Suppose you had a penny and a nickel. You hypothesized that if you flipped each of them 50 times, that there would be no difference (the Null Hypothesis) in the number of times that “Heads” came up with each coin. Suppose, in fact, that the penny came up heads 45 times and the nickel came up heads 52 times. There was a small difference. But in order to disprove the Null Hypothesis, the odds that this result occurred merely to chance would have to be less than 5%. It turns out that chance for this many flips covers from 22-78, so that even if the penny came up heads 22 times and the nickel came up 78 times, this would still be within the range of chance.

Now suppose the fellow next to you has 50 pennies, 50 nickels, 50 dimes and 50 quarters in a bag and dumps them all onto the floor. He knows nothing about statistics and makes no Null Hypothesis. He just wants to see what happens. He counts the pennies and determines that 33 are oriented as Heads, and 67 oriented as Tails. He “concludes” that pennies are twice as likely to come up tails as they are heads.

Clearly he has made an error in logic.

Normally, a reputable scientific publication such as Annals of Internal Medicine would show greater restraint in choosing to publish such a poorly designed study. However, the prospect of losing this study to another professional publication which would predictably send their summary to the nation’s newspapers and bring greater notoriety to a journal with an already declining readership was too much for them to bear. They succumbed to temptation.