Are peer-reviewed studies a bunch of crap?

[snailman2102]

I'm not sure where to post this, so feel free to relocate. I chose ADD since I suspect that people here are more likely to be familiar with the scientific literature.

A little background....I am a student of the health professions and have lately been learning about the treatment guidelines for various common diseases. These guidelines are based off of the evidence provided in various peer-reviewed studies. For example, suppose a "well-designed" study is published that shows that diabetics with well-controlled blood glucose have fewer myocardial infarctions. Well soon after this study is in print, all the doctors will go and change their practice: they will start aggressively controlling the blood glucose of all their diabetic patients. But how much does a study really prove? Regardless of how "well-designed" it is? It is very common that after a set of treatment guidelines is established, a new study will come out that proves THE OPPOSITE, for example that aggressive lowering of blood glucose increases mortality of diabetic patients.

Personally, the more I learn about the health professions, the more TERRIFIED I become of ever having to stay in a hospital. I advise anyone here who is being treated for any medical condition to do his/her own research and not blindly adopt the treatment suggested by your physician.

Thoughts?

 

[nuke]

The answer is: no, not really. When an unexpected finding is acknowledged on a smaller sample that may impact the mortality of a specific disorder, the consensus of the medical becomes "perhaps this has an impact on the mortality of this disorder." However, small sample sizes are well known among anyone who has taken a course in statistics to be prone to error, so follow-up studies are almost always conducted to find out if whatever hypothesis that was originally tested holds true for a larger population (hopefully one that is large enough to be comparable to the entirety of the population). After several of these studies are conducted, they are then placed into their own peer-reviewed meta-analysis studies, and when the meta-analysis can confirm these hypotheses, it becomes general medical consensus (though some doctors will still dispute anything).

Exceptions most often occur when money is involved, for instance in clinical trials for pharmaceutical companies where a company will downplay or not release studies that reveal results which may harm the ability of the company to sell these products. However, if you find something in a medical textbook, I would be willing to be dimes to dollars that it isn't far from the truth.

 

[Holy_cow]

^ Nice, thorouh answer to the original question. Before anything becomes a generally accepted treatment, there are many and large studies with the statistical power to prove if a treatment/drug is effective. But it should also be mentioned that this wasn't always so. In the old days if it worked at 30 out of 50 patients if might have become an accepted mainstay treatment.

 

[snailman2102]

Okay, I agree, but I think I failed to articulate the main idea of my question.

It is true that there are some conditions and drugs that have been around for awhile and have been studied extensively, with various meta-analyses and Cochrane reviews written about them. And yes, of course, these results are as reliable as we can hope for, given our current understanding of science.

But consider this example. Antiarrhythmic drugs (AADs) such as flecainide, quinidine, and propafenone had for several decades been administered to control arrhythmias. They work well for this purpose and there are many studies around that prove this, and many practitioners who readily used AADs to control arrhythmias in many patients. Within the past five or ten years, however, new well-designed studies have come out that looked at a different, and much more important endpoint--patient mortality, instead of symptomatic relief. These studies revealed that most AADs, despite reducing symptoms in many cases, actually significantly *increase* mortality. Further investigation into the mechanisms of AAD action revealed the previously-unknown fact that these drugs actually induce new types of arrhythmias in patients. Currently, AADs are thought of as "poisons with therapeutic side effects" and used with great care.

So here is my point. We are not even close to having a complete understanding of many very common diseases and we have NO IDEA how many commonly prescribed drugs work, or why they cause the side effects they do. Thus, even if we design a large, reproducible study that yields statistically significant results, how can we be sure that the study is measuring the appropriate endpoint or even asking the right question?

 

[snailman2102]

Peer review doesn't mean much to me, but could be valuable for a researcher in refining his postulates to minimally offend the reigning paradigms.

Yes, exactly. It seems to me that the scientific literature reflects the current scientific bias, and may or may not be evidence-based or at all appropriate for creating ideal therapeutic guidelines.

 

[nuke]

But then, what would you use to create ideal therapeutic guidelines? There's not really much else to go on than extensive amounts of empirical evidence. Mistakes do happen, but not nearly as bad as ones made in the beginning and middle of the last century (thorotrast and thalidomide, to name a few).

 

[BilZ0r]

I wrote a big reply, but the stupid fucking piece of shit asshole phpBB fucked out when I submitted it.

What I said was essentially thus. You're problem isn't with peer review per se , but with how clinical trials go from inception to practice.

1) Studies are only as good as what is actually done. This means both that if people do a study in a poor way its results may not be legitimate. Just because a study got through peer review doesn't mean it's perfect, or even that it's good enough (but one hopes the better the journal the better the review, but that is not really the case). Also sometimes what people say was done in a study was not what was ACTUALLY done. Usually this is due to a mistake, and not purposeful deception, but that happens too.

2)The statistical tests that people use to say whether the intervention worked or not, are by their very nature not accurate 100% of the time. Usually they are set up to only fail 5% of the time. Moreover, apply the wrong statistical test to a dataset can yield an incorrect result; and seeing as it takes a lot of knowledge in statistics to know whether you should use an ANOVA or a multiple logit regression, (and the people who review trails are usually doctors/scientists not biostatisticians) this mistake happens very often.

3)People often interpret legitimate studies incorrectly. Classically, it goes something like a study shows that, say, eating 100kg of tomatoes every year significantly increases your lifespan. It might be that it only increases your lifespan on average by 6 months, i.e. insignificantly. Yet people read this as "Eating tomatoes makes you live longer". Indeed, a more relevant example could be a drug which reduces cholesterol. It might have a statistically significant effect, but if it only reduces cholesterol in a high cholesterol group by 10%, those people are still fucked.

A more concerning trend is how clinical trials are using biomarkers for disease rather than the disease itself. Imagine a new drug which is supposed to reduce myocardial infarctions (MIs) in an at risk group. Instead of measuring actually MIs (which would take years, maybe a decade to get a statistically significant result), the company just measures a blood protein which correlates with MI in a total population (like c-reactive protein). They conclude, less C-reactive protein, less MI, and that the drug works, even though there hasn't been a single heart attack in their study. Currently the FDA accepts this.