[sciborg2]

"Everything We Know Is Wrong"

...Again and again, researchers are finding the same things, whether it's with observational studies, or even the "gold standard" Randomised Controlled Studies, whether it's medicine or economics. Nobody bothers to try to replicate most studies, and when they do try, the majority of findings don't stack up. The awkward truth is that, taken as a whole, the scientific literature is full of falsehoods.

Jolyon Jenkins reports on the factors that lie behind this. How researchers who are obliged for career reasons to produce studies that have "impact"; of small teams who produce headline-grabbing studies that are too statistically underpowered to produce meaningful results; of the way that scientists are under pressure to spin their findings and pretend that things they discovered by chance are what they were looking for in the first place. It's not exactly fraud, but it's not completely honest either. And he reports on new initiatives to go through the literature systematically trying to reproduce published findings, and of the bitter and personalised battles that can occur as a result...

[Kellais]

Quite the polemic paper, eh?!

On the other hand i do sometimes wonder. Just look at all the contradictory results in something like "what food/beverages are healthy/not healthy". Sometimes they say "Coffee is unhealthy" sometimes they say "Coffee is healthy" etc etc . I guess in such studies it is always important to know where the money to finance the study came from, right?!
On a more serious note though, this is indeed a problem...underpowered statistics, studies that have only been done once or twice (so are not proven to be reproducable etc) are not that reliable. But as research is, unfortunately, still totally dependant on money, and money is most often only given to study-areas that are useful to the big companies, i guess we will have the problem for the time being (especially in areas that are pure science with no direct application for the economy).

Although, as i hinted at, money being around is not always the problemsolver one thinks....if you take the money of entity X they kind of want to...influence...what you find. Which begs the question if we really want to have such "studies".

As the saying goes: "Only believe in the statistics you falsified yourself..."

[The Sharmat]

These biases would effect government funded research as much as private. If this paper is to be believed, neither of them have a vested interest in disproving anything, so replicating results is kind of an anemic part of the division of labor in science. In a way it can be viewed as spending more money for the privilege of knowing you wasted your money, and an irrational human being that's really loss averse is not gonna want to explore that option. Plus, spending your time attempting to reproduce the results of others' studies just isn't particularly glamorous work that most scientists aspire too.

That said this article isn't even linked in the BBC article so I can't study it in detail to find out if it's accurate or if it's just another one of the headline grabbing articles it maligns. Science reporting sucks.

[Kellais]

Well i guess the onus of reproducable results would lie on the group that first attempts a study. So it's not the task of a new group to verify that. At least imo. Although if you can reproduce an earlier study and then add to it, that's surely not a bad thing.
I agree though, it's not very glamorous. But i guess if you are only in research for the glamour, than you're in the wrong place anyway

[The Sharmat]

Well i guess the onus of reproducable results would lie on the group that first attempts a study. So it's not the task of a new group to verify that. At least imo.

There is a clear conflict of interest there. I suspect if you didn't place the burden of reproducing the results on another group of people, you'd mysteriously very rarely see the results fail to be reproduced.

[Kellais]

Well, if you have to publish your data, you can't cheat. So i guess one could avert that problem. But i admit, it's not without work. And yes, if you just can say "yes, we were able to reproduce our results" without giving proof, it will be easy to do so.
But in the end, if you are deep into real science, you can't hide for long that you faked your results...because you will just not get anywhere. Or the people who come after you to work on the same or similar stuff will uncover your lie.

It's kind of sad that there are branches of science/research where there are people who feel the need to cheat (i guess that's what you're implying, right?). But i guess it's the usual problem...where there's real money, there are cheaters/liars.

[The Sharmat]

There have been a number of famous cases of faked data, in particular I remember some doctored Stem Cell research coming out of South Korea.

If we're not striving to replicate data at an especially fast clip, there could be many more erroneous or outright false studies not being detected.

[Alia]

Well, as for underpowered studies or messing with statistics, basically that's what peer review process is for. The reviewers should be able to recognize these flaws and stop the study from being published in the journal. Unfortunately, there are many pseudo-scientific journals that would publish anything, as long as the authors pay. And peer review is not without faults (Sokal comes to mind).

[sciborg2]

I posted this in a prior thread, makes you wonder how much of science is just pseudoscience:

Why Most Published Research Findings Are False

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

[Royce]

As Jed Mckenna says, science should come with a disclaimer.  WARNING: The scientific findings contained herein are based on the uncritical acceptance of consensus reality as true reality, and must therefore be viewed in the same light as mythology, folklore, superstition and religion.

Scientists, due to years of highly specialized training, are probably the most narrowly focused and effectively indoctrinated people in the modern world.

[Kellais]

Oh boy!
Is that your view, Royce? Maybe that'd be worth an own thread.

Suffice it to say i have strong reservations with a lot of the parts of your post. Not the least of which are putting science in the same fantasy land as superstition and religion. And also thinking that scientists are narrowly focused and/or indoctrinated just because they are highly trained.

[Royce]

Oh boy!
Is that your view, Royce? Maybe that'd be worth an own thread.

Suffice it to say i have strong reservations with a lot of the parts of your post. Not the least of which are putting science in the same fantasy land as superstition and religion. And also thinking that scientists are narrowly focused and/or indoctrinated just because they are highly trained

Lol:)  Is it my view? No not really. I borrow views all the time, change them around like underwear really. If you don`t you are a fundamentalist!:)

If I am 100% honest, and 100% skeptic, "science" is also a matter of belief. It is consensus, just taken a bit more seriously then religion. It is not a popular view, but it is true, if you are honest with yourself.

That being said, I am not anti-science or anything. It works, it definitely has its uses, but that does not make it truth. Just sayin:)

[Kellais]

Well i also look at views of other people But i am not sure one has to borrow them to not be a fundamentalist

I am not sure that science and religion are on the same level of "belief". Religion is nothing but belief. Science is more substantial than that. Sure, depending on which field you look at, a lot of it is not something you can touch for real, so to speak. But it has real world applications that are reproducable and demonstrably true.
And the thing with the truth...well we talked about that at great length in another thread, didn't we As long as there is no real difference for almost everyone on this planet if it is "real truth" (TM) or just "something like truth", it doesn't matter...and we can just call it truth. Even if technical definitions of quantum physics or neuroscience seem to have theories for the contrary, usefull is what finds an application in our lives, be they real or imaginative dreams of...whatever

[Bolivar]

My fiance is in medicine and I often help her with literature reviews and the like. Most recently, we looked at studies assessing the benefits of vitamin supplements and it was shocking to see just how fraught with confirmation bias even a seemingly non-controversial topic could be.

A friend of mine who does business analysis pointed out that most scientists are not great statisticians, so their ability to distill the significance of their research is severely handicapped. I feel like the gap between our abstract ideas and the actual data it is supposedly based on is getting wider, creating a culture of certainty that is as unfounded as it is often pretentious.

[Royce]

Well i also look at views of other people But i am not sure one has to borrow them to not be a fundamentalist

I mean it literally though. I have no view or opinion which I can say is "mine". All my views are borrowed, always has been and always will be. I think that is the case for everyone. Unless you invent a new language, this must be the case:)