1
Philosophy & Science / The Flawed Reasoning Behind the Replication Crisis
« on: August 08, 2019, 01:23:10 pm »
Good article from nautil.us highlighting flawed statistical thinking
http://nautil.us/issue/74/networks/the-flawed-reasoning-behind-the-replication-crisis
...
And this part in particular will likely make me even more unpopular on this forum
http://nautil.us/issue/74/networks/the-flawed-reasoning-behind-the-replication-crisis
Quote
Here are three versions of the same story:
1. In the fall of 1996, Sally Clark, an English solicitor in Manchester, gave birth to an apparently healthy baby boy who died suddenly when he was 11 weeks old. She was still recovering from the traumatic incident when she had another baby boy the following year. Tragically, he also died, eight weeks after being born. The causes of the two children’s deaths were not readily apparent, but the police suspected they were no coincidence. Clark was arrested and charged with two counts of murder. The pediatrician Roy Meadow, inventor of the term “Munchausen Syndrome by Proxy,” testified at the trial that it was extremely unlikely that two children from an affluent family like the Clarks would die from Sudden Infant Death Syndrome (SIDS) or “cot death.” He estimated the odds were 1 in 73 million, which he colorfully compared to an 80:1 longshot winning the Grand National horse race four years in a row. Clark was convicted and sentenced to life in prison. The press reviled her as a child murderer.
...
Quote
The mathematical lens that allows us to see the flaw in these arguments is Bayes’ theorem. The theorem dictates that the probability we assign to a theory (Sally Clark is guilty, a patient has cancer, college students become less theistic when they stare at Rodin), in light of some observation, is proportional both to the conditional probability of the observation assuming the theory is true, and to the prior probability we gave the theory before making the observation. When two theories compete, one may make the observation much more probable, that is, produce a higher conditional probability. But according to Bayes’ rule, we might still consider that explanation unlikely if we gave it a low probability of being true from the start.
So, the missing ingredient in all three examples is the prior probability for the various hypotheses. In the case of Sally Clark, the prosecution’s theory was she had murdered her children, itself an extremely rare event. Suppose, for argument’s sake, by tallying up historical murder records, we arrived at prior odds of 100 million to 1 for any particular mother like her to commit double infanticide. That would have balanced the extreme unlikelihood of the observation (two infants dying) under the alternative hypothesis that they were well cared for. Numerically, Bayes’ theorem would tell us to compare:
(1/73,000,000) * (99,999,999/100,000,000) vs. (1) * (1/100,000,000)
We’d conclude, based on these priors and no additional evidence aside from the children’s deaths, that it was actually about 58 percent likely Clark was innocent.
Quote
The problem, though, is the dominant mode of statistical analysis these days isn’t Bayesian. Since the 1920s, the standard approach to judging scientific theories has been significance testing, made popular by the statistician Ronald Fisher. Fisher’s methods and their latter-day spinoffs are now the lingua franca of scientific data analysis. In particular, Google Scholar currently returns 2.85 million citations including the phrase “statistically significant.” Fisher claimed signficance testing was a universal tool for scientific inference, “common to all experimentation,” a claim that seems borne out by its widespread use across all disciplines.
And this part in particular will likely make me even more unpopular on this forum
Quote
Just a few of the other casualties of replication include:.
- The study in 1988 by Strack, Martin, and Stepper on the “facial feedback hypothesis:” when people are forced to smile, say by holding a pen between their teeth, it raises their feeling of happiness.
- The 1996 result of Bargh, Chen, and Burrows in “social priming,” claiming, for example, when people are exposed to words related to aging, they adopt stereotypically elderly behavior.
- Harvard Business School professor Amy Cuddy’s 2010 study of “power posing:” the idea that adopting a powerful posture for a couple of minutes can change your life for the better by affecting your hormone levels and risk tolerances