14 December 2010

Deceptive numbers

The New Yorker has a fascinating article [$] on scientific method that asks a basic question: Why do the statistically significant results in pathbreaking publications disappear when studies are repeated by the same scientists using the same protocols? The simple answer is "reversion to the mean." Since studies with exceptional results get published while studies that find nothing do not. This can happen when results are indeed exceptional ("we discovered that Prozac makes you happy") or because a trial -- even a double-blind trial with a large sample size seemed exceptional when it was really just chance ("Prozac made this group happy, so let's approve it as a drug even though it will not make other groups of people happy"). These are NOT experiments in the social sciences (which are really hard to do well), as much as medicine, biology, zoology, etc. The implications are enormous: We may be making policies and regulations that are based on statistical flukes, not on the real world.

"The extraordinary implication... is that a lot of extraordinary scientific data are nothing but noise... Just because an idea is true doesn't mean it can be proved, and just because an idea can be proved doesn't mean it's true."

Bottom Line: Buyer beware. Useful solutions are likely to come from techniques that are hundreds of years old, not the results of a few double-blind trials.

2 comments:

Eric said...

Having done science for decades now, I would say that the real story is much more complex than the New Yorker paints it. Some results disappear. Almost always there is a reason. Most results do not disappear.

The rate of disappearance of results is very dependent on the field of study. In a field composed of a few basic assumptions, such as parts of physics, results never disappear. In fields such as social psychology where both the experiment and the conditions of the experiment may be less well defined, results are more likely to be invalidated or at least questioned.

Justin Kraus & Joy Kim said...

The STS (Science, Technology, and Society) folks have been saying this stuff for a long time. I wish more people would pick up on it. Read Bruno Latour, Steven Shapin, Donald Mackenzie, or John Laws. There is a lot more going on with Science than the simple application, and replication, of some idealized methodology.