Reblogged: A Band-Aid for Science

[Gus Van Horn blog]

Here's a sign of the times: The man who once published a paper titled, "Why Most Published Research Findings Are False", and others have formed a new institute to police bad science. In addition to bogus results, the institute (METRICS, for "Meta-Research Innovation Centre at Stanford") will also consider the problem of wasted effort:

... A recent series of articles in the

Lancet

noted that, in 2010, about $200 billion (an astonishing

85% of the world's spending on medical research) was squandered on studies that were flawed in their design, redundant, never published or poorly reported

. METRICS will support efforts to tackle this extraordinary inefficiency, and will itself update research about the extent to which randomised-controlled trials acknowledge the existence of previous investigations of the same subject. If the situation has not improved, METRICS and its collaborators will try to design new publishing practices that discourage bad behaviour among scientists. [bold added]

While this effort is laudable, I think it will fail, because I think many of these problems are ultimately due to government funding of science. Indeed, it reminds me a little of recent efforts to address the problem of the government training too many new scientists in certain fields -- such as by adding a whole new training program to their terminal degrees.

Set aside, for the sake of argument, the whole question of whether the government should be taking money forcibly from some citizens for any reason. What we are seeing in science, as with many other areas of the economy, is a vast amount of money being poured in to an industry by a political class whose primary motivation isn't the discovery of truth (rather than flowing there due to market forces) and creating a whole slew of artificial, perverse incentives. When these perverse incentives guide individuals, it should come as no surprise that it is common to see examples of money being wasted, be it in the form of sloppy studies, redundant work, or over-training. It may be possible to mask or slightly alleviate the symptoms for a time, but the disease goes merrily on.

-- CAV

Link to Original

[softwareNerd]

Journals are supposed to use "referees", who are experts in the field, to do QA on submissions.  There will always be problems, but the system itself seems to be broken... which indicates that the incentives to publish are much stronger than the incentives to insist on better quality. 

 

The whole global warming boondoggle is a good example of bad science. Sokal's paper is a hilarious example -- but this is black humor that comes back to hurt us all. On nutrition advice, for instance, one study that says one thing, and then a study that says the opposite. On DDT, politics triumphed over truth. 

 

Meanwhile, the press has failed in its own QA mission. Since universities have drilled the idea that knowledge is subjective, reporters think objectivity consists mostly in presenting different opinions. The reporter's own mental evaluation is only a small part of objectivity. And, a cynical attitude means that entertainment also trumps truth (man bites dog), as we see from the crap on the History Channel and the Discovery Channel. As a result, when one side pushing a story -- e.g. a researcher with a paper -- the counter-argument gets scant attention if the claimed result makes good copy. 

 

Perhaps if some private organization built a reputation for independent QA, the press would then turn to it occasionally for a second opinion. 

[Nicky]

Looking through the list of hoaxes similar to Sokal's on wikipedia, I found an even scarier one:

http://en.wikipedia.org/wiki/Rosenhan_experiment

[softwareNerd]

Looking through the list of hoaxes similar to Sokal's on wikipedia, I found an even scarier one: http://en.wikipedia.org/wiki/Rosenhan_experiment

Interesting. I can understand being diagnosed as insane based on feigned symptoms. Perhaps now there are some types of scans etc., but self-reporting is presumably still the biggest factor. If I tell a doctor that my dead wife is sitting in the chair beside me and making comments about the doctor's hair, it seems reasonable for him to think I am hallucinating rather than lying.

In the wiki write-up, the symptoms hardly seem to justify hospitalization, and even if they were hospitalized in order to be observed, one would imagine they'd be released after the symptoms did not show for a few days.

I recommend the movie "Changeling", which is based on this topic.

 

From conversations with a friend, it may not be an issue today. He told me about an older member of the family who hallucinated and where the family all thought he should be in a hospital. However, the few times he went, he would check himself out in a few days time, and the law prevented the hospital staff from keeping him -- he was really no danger to others, but they were scared he would wander off, or do something dangerous.

[Nicky]

The "holding people for bogus psych reasons" industry is still thriving in the US. I have a friend who was taken to hospital for alcohol poisoning (not by cops, by her friend, who was worried about her physical well being). While she was drunk, she had a conversation with a phychiatrist that she has no memory of. Based on it, she was held for several days and medicated against her will, supposedly for being a danger to herself. The kicker is, she wasn't restrained or put under supervision. From talking to the nurses, she found out that she was free to leave at any time, and no one had the means to physically stop her, except for the fact that leaving would've been illegal (and resulted in minor, but costly legal issues, not doubt).

Obviously not an arrangement that would've helped her if she really was suicidal. But one that allowed the hospital to charge her insurance for her full stay.

[Robert Baratheon]

That story makes no sense. In what sense was she "held" and "medicated against her will" in light of the fact that she wasn't restrained and the nurses told her she was "free to go"? What law supposedly made it "illegal" for her to leave?

 

I wouldn't accept a second-hand account from someone who admits she was so intoxicated she can't remember key elements of what happened.