John Ioannidis paints a picture of a vast hive of researchers all pushed to publish short papers that are mostly a waste of time. The design is bad, the results useless (even when meta-collated with other badly designed studies). Basically, humankind is pouring blood, sweat and tears into spinning wheels in medicine — just paper churn. Most papers will never help a patient.
Ioannidis wants rigor – full registration before the study, full transparency afterwards, fewer studies over all, but with better design. Astonishingly, fully 85% of what is spent on clinical trials is wasted. It’s really a pretty big scandal, given that lives are on the line. I can’t see the media or pollies joining the dots. Imagine how many quality life-years are being burnt at the stake of the self-feeding Science-PR-Industry.
And this is clinical medical research, where standards are higher than in many other scientific areas and where there are easily defined terms of success unlike “blue sky” studies. Ioannidis doesn’t say it directly, but his description of the effect current funding has (which is almost all government based) almost guarantees that researchers will be wasting time in the paper churn — fast, short papers of little importance, that may even be false, but even if true are useless, insignificant. This is what happens when science is controlled by a government monopsony. The aim is the press release, not the patient.
Science can’t be done by an indexed formula or citiation score. Money can’t be spent wisely that way either. Someone needs to be responsible.
How much of climate research is a waste of money? A lot more than in clinical medicine.
Jo
Conclusion
Overall, not only are most research findings false, but, furthermore, most of the true findings are not useful. Medical interventions should and can result in huge human benefit. It makes no sense to perform clinical research without ensuring clinical utility. Reform and improvement are overdue.
The sheer size of the waste and the industry — 85% of a million papers
There are many millions of papers of clinical research—approximately 1 million papers from clinical trials have been published to date, along with tens of thousands of systematic reviews—but most of them are not useful. Waste across medical research (clinical or other types) has been estimated as consuming 85% of the billions spent each year [1]. I have previously written about why most published research is false [2] and how to make more of it true [3].
Clinical research remains extremely expensive, even though an estimated 90% of the present cost of trials could be safely eliminated [26,27]. Reducing costs by streamlining research could do more than simply allow more research to take place. It could help make research better by reducing the pressure to cut corners, which leads to studies lacking sufficient power, precision, duration, and proper outcomes to convincingly change practice.
The problem is the research funding:
Current research funding incentivizes small studies of short duration that can be quickly performed and generate rapidly publishable results, while answering important questions may sometimes require long-term studies whose financial needs exceed the resources of most currently available funding cycles.
One suggestion to improve research funding:
One to two percent of the sales of blockbuster drugs diverted in such a pool [52] could earmark ample funding.
Snippets of interest from the full paper:
Transparency (Trust)
Utility decreases when research is not transparent, when study data, protocols, and other processes are not available for verification or for further use by others. Trust is also eroded when major biases occur in the design, conduct, and reporting of research.
Only 61% of trials published in clinical journals in 2010 had been registered [30], and rates are much lower for nonregulated interventions [31] (e.g., 21% and 29% for trials published in psychological or behavioral [32] and physical therapy [33] journals, respectively). Only 55/200 (28%) of journals that publish clinical trials required trial registration as of 2012 [34]. Few full protocols are registered, analysis plans are almost never prespecified, and the full study data are rarely available [35].
Focusing on major journals.
Some clinicians prefer to read only research published in major general medical journals (The New England Journal of Medicine, The Lancet, BMJ, JAMA, and PLOS Medicine). However, these journals cover a tiny minority of published clinical research. Out of the 730,447 articles labeled as “clinical trial” in PubMed as of May 26, 2016, only 18,231 were published in the major medical journals. Most of the articles that inform guidelines and clinical practice are published elsewhere.
Getting “free” research from students, volunteers and trainees is part of the problem, he says:
Clinical Research Workforce and Physicians
The clinical research workforce is huge: millions of people have coauthored at least one biomedical paper, and most have done so only once [48]. Students, residents, and clinical fellows are often expected to do some research. This exposure can be interesting, but trainees are judged on their ability to rapidly produce publications, a criterion that lends itself badly to the production of the sort of large, long-term, team-performed studies often needed to inform us about health, disease, and health care. Such researchers can become exploited as low-paid or volunteer personnel [49], and an untrained, noncommitted workforce cannot produce high-quality research. Other perverse recipes in clinical research include universities and other institutions simply asking for more papers (e.g., least publishable units) instead of clinically useful papers and clinical impact not being a formal part of the publication metrics so often used to judge academic performance. Instead of trying to make a prolific researcher of every physician, training physicians in understanding research methods and evidence-based medicine may also help improve the situation by instilling healthy skepticism and critical thinking skills.
h/t Kip Hansen at Judith Curry
REFERENCES
John P. A. Ioannidis (2016) Why Most Clinical Research Is Not Useful, PLOS Med Published: June 21, 2016 http://dx.doi.org/10.1371 /journal.pmed.1002049
Ioannidis JP. Why most published research findings are false. (2005) PLoS Med. ;2(8):e124. pmid:16060722 doi: 10.1371/journal.pmed.0020124 ..View Article. PubMed/NCBI
Ioannidis JP. (2014) How to make more published research true. PLoS Med.;11(10):e1001747. doi: 10.1371/journal.pmed.1001747. pmid:25334033.View Article. PubMed/NCBI .
As long as academics are awarded through point systems based on number of peer-reviewed articles published quantity >>> quality. My handful of years in academia were enough to see how warped system the system is. Like all humans, scientists will game the system to achieve the rewards at the end, be it government funding, super-long CVs, position etc. For some reason many people are impressed with long lists of publications under a persona’s name.
As Einstein said, it would only have taken one paper to prove him wrong, but then he was focused on quality, not quantity.
160
Yes, it’s the old “publish or perish”* system. Churn out those papers if you want to keep your lecturing job, if you you are going to have a hope of tenure (heavenly choir sings), and especially if you want to go to international conferences in Cancun with your three young, female, research assistants.
(*I’m told that in Canada it is “publish or prairies”, which is even scarier.)
21
And, for the purposes of disclosure, no, I never got even close to a tenure-track position, let alone tenure.
But I’m not bitter about it.
Not at all.
No bitterness here.
Don’t mind in the least.
No grinding of teeth in envy of those who made it.
Perish the thought.
Etc.
30
“Basically, humankind is pouring blood, sweat and tears into spinning wheels in medicine ”
How far we’ve descended from the pinnacle of medical achievement.
http://www.jameslefanu.com/books/the-rise-and-fall-of-modern-medicine-introduction
70
For the tip of the very, very large Iceberg of scientific incompetence and plain straight out F****!
Retraction Watch and the listed retractions below from July 6th to,July 9th
————
eg; Authors retract study that found pollution near fracking sites [ more on this on the GWPF forum; ANTI-FRACKING SCIENTISTS QUIETLY RETRACT PAPER DUE TO BAD DATA]
.
Authors who lost paper linking fecal transplants to obesity have another retracted
.
PLOS ONE pulls malaria study for “inappropriate manipulation” of figures
.
Five more retractions for biologist with funding ban brings total to six
.
Crow’s feet filler study omitted pharma funding, gets retracted
.
Heir claims part of review about political scientist is defamatory, journal partially retracts
.
You’ve been dupe’d (again): Do these data look familiar? They are
.
Botanist pair’s paper retracted, others questioned on PubPeer
——–
And quite a few years worth of similar retractions in “Retraction Watch’s” Older entries
100
The pressure to publish lots and lots of small papers, each with little worth, was what was obviously going on when I was at Uni 30 years ago. Thats what got people promotions and money. Research had to be in the trendy subject and use the right buzz words. If it didnt the academics didnt get their mortgages paid. I shudder to think how bad things have become since then.
I saw how political “science” was. That was pre-global warming nonsense. I was sceptical about what the next Buzz words in science were going to be. Then it came along worse than I anticipated.
I worked as a medical rep some years after being at Uni. I remember reading the papers the company gave us to hand out. They were rubbish, I didnt hand them out to doctors. For example, how can you take seriously a trial with a sample size of 9 ?
They were rubbish studies, just as described here.
110
The MMR vaccine scare was started by a paper with a sample size of 15. That’s the ideal sample size for modern science!
52
Sometimes a small sample is sufficient. Maybe not to establish the proof or otherwise of the hypothesis but as a first step.
The children were suffering from disorders that were only evident post MMR vaccination. The research was done by a competent team. It was found that there did appear to be the possibility of a causal relationship and further research was recommended.
There’s a lot more to this story. A link between the MMR vaccine and autism is no longer in question and is the subject of the recently released movie “VAXXED”.
And is the reason the MMR vaccine is now banned in Japan.
41
Jo, you ask – “How much of climate research is a waste of money?”
I would suggest the magic figure of 97% might apply.
100
85% its a funny thing but today I came across the same number in a report on Health Care in Australia. To make up 85% of an average specialists consultations you only need to include 5 different problems. BEACH study year after year has shown that to cover 85% of the consultations a GP has to sort out you would have to face 167 different problems. BEACH has now been discontinued by the Turnbull Government as part of its reduction in Medicare expenditure. Bit like putting on a blindfold to save on the cost of spectacles really.
We do an annual research project in my practise because my students have to complete a research project to qualify for the year. I always insist we do something that will improve the quality of care we provide, and its never the sort of stuff we read in journals, most of which report studies done in specialist clinics. These papers are always presented but almost never published as that is not the primary objective. But sitting in on a presentation evening is fascinating and absence of evidence is a common apology given by the student, unaware that absence of evidence is as critical as evidence itself.
30
The trouble is when medical researchers do more rigorous studies with many thousands of subjects , the results can often be embarrassing to the so called experts and their paymasters like big pharma , take this recent research that looks at the effects of cholesterol lowering statins on people over sixty , unfortunately for the profits of the pharmaceutical companies the results were (rather quaintly) said to be the ” inverse ” of what they hoped for ; https://www.sciencedaily.com/releases/2016/06/160627095006.htm
70
That analysis of government funding wasted on useless medical research does not include a whole field of quackery, Climate Medicine.
https://rclutz.wordpress.com/2016/07/10/climate-medicine/
40
Once you have read the link that I posted above , just sit back for a few moments and consider the extraordinary implications that the study suggests , statins are for reducing so called LDL or ” bad ” cholesterol , but they find that higher levels of that type of cholesterol actually help extend life spans , so taking statins to reduce cholesterol means more premature deaths , no wonder it was published in a relatively obscure journal , many millions of people have been taking statins , how many have died prematurely because of them ? the trouble is most institutionalized scientists go along with the consensus because the big pharma companies that make billions on statins fund them either directly or indirectly .
50
There have been a lot of changes to recommendations about diet based on some of these studies. Butter is bad, margarine is good, mono saturated oils are good, too many eggs are bad because of their cholesterol content, and so on. Difficult to know what to eat or drink apart from doing things in moderation.
30
Whenever I see “eggs are bad,” I wonder, “What do they mean ‘eggs’?” Not all eggs are alike.
I always seem to ask the question that “scientists” are too dumb to ask: What are you feeding the chicken that’s producing the eggs?
40
Its just another case of the great LPU (Least Publishable Unit) Bubble that opened my eyes to what academia and much of the ‘research’ world really were. Munchhausen Syndrome with Tenure
20
Climate science has become rootless, divorced from its very subject. You can’t even get a temperature researcher to notice clouds, because the whole silly min/max fiddle relies on ignoring cloud. (Not that anyone should care if things have warmed a trice overall, any more than one should worry about the dribble of sea level rise that’s been going on since the 1700s.)
The human and natural history of our Holocene epoch has never been refuted, just minimised or ignored. This is done to maintain the pretense of a “stable” climate which has never existed. The marked changes of very recent millennia, especially the almighty heave out of the Younger Dryas and into the Optimum (accompanied by major sea level rises and events such as the filling of Bass Strait and the Storegga Slides) are de-emphasised or just filed away like faded sepia photos in a forgotten drawer.
So many people think these changes occurred millions or hundreds of thousands of years ago…and our climate experts are in no rush to explain that they occurred over just two or three thousand years and within the period of human settlements and even the first towns. Want a problematic climate change? Try a Bond Event like the cooling that smashed up Mesopotamian and Egyptian civilisation around 2200 BC. Or try any of the marked cooling events like the Migration and LIA periods. Not that long ago!
Climate has changed a lot, has changed quickly, has changed all the time, and it has changed in both cyclical and linear ways. The problem with climate science is that those most vocal about climate change will not accept climate change except as a political confection.
40
Mosomoso; I just love that phrase you’ve used . . ‘political confection’ It is so true and most apt; the real facts and issues of climate science are just not understood by so many. They prefer a simple and easy argument rather than try to understand/get to the truth which is more problematic.
GeoffW
20
Whoa! Let’s not get too depressed. Don’t forget that the human body and living organisms are unbelievably complex, but do consider the amazing developments that have taken place in relatively recent times.
Take myself as an example. I’ve recently been diagnosed with prostatic cancer. The prostate specific antigen (PSA) test was introduced for clinical use in 1986, and I’m very glad that suspicions were raised by my test result (and yes, I know the results aren’t always conclusive – but that’s the nature of human biology). The biopsy (for the benefit of anyone who might need this procedure in the future, it’s a painless affair, done under local anaesthetic) confirmed the presence of malignant cells. An MRI scan (introduced I believe in the 1980s and continually refined since) indicated that the tumour has not spread. Such scans provide valuable information for surgical planning. According to a book I bought on the subject, “the integration of radiographic and radiotherapeutic strategies to stage, select, and treat patients with clinically localised cancer has provided a major step forward in treatment of this disease”.
I’m currently on androgen blockers to stop the tumour growing, and to shrink the prostate so as to reduce radiation damge to surrounding healthy tissues when radiotherapy begins.
Yes, a lot of medical research doesn’t lead anywhere – but that’s to be expected. Been there, done that! Five years of solid, careful bench work on a vaccine-related project didn’t yield the hoped for results. So don’t be too hard on all those researchers!
Thanks to modern medicine, it’s expected that my cancer will be successfully treated.
70
Best wishes Carbon500. Consider looking up David Archibald and sulforaphane and capsaicin. (From broccoli and chilli). These two natural molecules act together on TNox which is expressed on cancer cells and trigger apoptosis — the natural cell death of cancerous cells. I’ve seen lab test results on prostate cells which looked promising, and as the nutrients are low risk the downsides appear small. Archibald was trying to get clinical trials done but the two together could not be patented, so no one would fund the research. It is exactly the kind of clinical trial the government ought to fund. They seem to shrink the prostate too.
60
Thank you very much, Jo. I’ve read the fascinating link you’ve kindly supplied, and I’ll get a copy of ‘Hormones with Harmony’ – it looks very interesting indeed.
Since retiring from the National Health Service here in England, I have to say I’ve missed science somewhat, despite having retired with a full set of hobbies. The man-made global warming issue has kept me occupied, but my recent diagnosis has given me the incentive to get back to my healthcare interests.
Despite my MRI scan showing no signs of disease spread, I certainly think it will be wise to take note of the information you’ve sent, read further and consider long-term dietary changes. I very much appreciate the biochemical explanations you’ve drawn my attention to. It wouldn’t be difficult to modify my diet, and what better incentive than keeping cancer at bay?
40
Thanks for that link Jo. I’ve just spent ages reading it and want to go back to it again. Fascinating. I’m way too late for bed now!
00
I work for a company (oil industry) which pays small, but not insignificant bonuses for publishing work, for presenting posters at conferences, and the like. They also count towards promotions as well. Other companies have similar incentives. The ‘default’ method to do this for anything related to engineering, software, and the like, is via an industry based society. In theory, this society has strict rules for submissions, but in practice, published content is often woeful. It starts with basic spelling errors and atrocious formatting, right up to blatant plagiarism, flawed data interpretation, and outright lies about results. Some material is nothing short of advertising material for new products, providing as much technical depth as the average junk-mail flyer.
That’s not to say that everything published is rubbish, because there is indeed some extremely high quality work being presented, and being presented very well, but it now takes a great deal of time and effort to sort the wheat from the chaff.
In theory, a percentage of these submissions are classified as ‘peer reviewed’, and indeed, that does very often lead to better quality material, but nonetheless, what was once a trustworthy and prestigious organisation has tacitly agreed to a massive reduction of standards, presumably to keep subscriptions and members, but also because the very people who end up in positions of authority in said society, have usually got there via publishing lots of papers (again, not all of them, but enough of them).
40
Interesting, after near 40 years in the oil industry and several papers and posters published over the years, I never saw any bonuses or promotions tied to publications, and I had not heard of other companies providing such incentives. Perhaps I didn’t work for the type of companies who paid out dollars and promotions, despite once being awarded a Best Paper at an industry conference.
Ciao
John
10
The fundamental problem is conflict of interest not irrelevant research. How does one fight the conflict of interest problem? Why is there propaganda? How can groups of good people do bad things?
An example to illustrate the nature of the conflict of interest problem is the astonishing increase in the prescriptions of very, very, dangerous ineffective psychoactive drugs. How is possible that dangerous ineffective psychoactive drugs are prescribed to millions and millions of people?
The general public’s beliefs concerning the mind and psychoactive drugs is primarily based urban legends, on industry created propaganda. There are almost a hundred different books concerning the scandal. Whitaker’s book is one the best written.
Whitaker’s book is available in most libraries. Jump to the case story, in Whitaker’s book, of 10 year old Jasmine how is ‘treated’ for anxiety concerning an overnight camp due to concerns over bed wetting. Jasmine is a dubbelganger for mine daughter Tess.
https://www.amazon.com/Anatomy-Epidemic-Bullets-Psychiatric-Astonishing/dp/1491513217
Anatomy of an Epidemic: Magic Bullets, Psychiatric Drugs, and the Astonishing Rise of Mental Illness in America
“This plague of disabling mental illness has now spread to our children, too. In 1987, there were 16,200 children under eighteen years of age (William: In the US) who received an SSI (William: SSI social security disability insurance) payment because they were disabled by a serious mental illness.
But starting in 1990, the number of mentally ill children began to rise dramatically, and by the end of 2007, there were 561,569 such children on the SSI disability rolls. In the short span of twenty years, the number of disabled mentally ill children rose thirty-five fold (William: and the prescriptions of psychoactive drugs to children has also risen by 35 times). Mental illness is now the leading cause of disability in children, with the mentally ill group comprising 50 percent of the total number of children on the SSI rolls in 2007.”
Comment: The number of adults in the US on social disabled insurance for mental ‘illness’ has increased by a factor of 4 for the same period which roughly matches the increase in prescription of psychoactive drugs to adults in the US over the same period.
https://en.wikipedia.org/wiki/List_of_largest_pharmaceutical_settlements
Healy’s book explains in detail how research is manipulated to hide side effects, to push drugs that are in some cases no more effective than placebos and less effective that older drugs that are no longer patent protected.
Pharmageddon
https://www.amazon.com/Pharmageddon-David-Healy/dp/0520270983/ref=pd_sim_14_9?ie=UTF8&dpID=51NRLBEXsDL&dpSrc=sims&preST=_AC_UL160_SR105%2C160_&psc=1&refRID=N9XGMY2JH6ZH09JPXX8D
Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients
https://www.amazon.com/Bad-Pharma-Companies-Mislead-Patients/dp/0865478066/ref=cm_cr_arp_d_product_top?ie=UTF8
http://www.nybooks.com/articles/2009/01/15/drug-companies-doctorsa-story-of-corruption/
20
Research, for which taxpayers pay,
Is produced by the ton every day,
Mostly useless additions,
Not improving conditions,
For mankind in any real way.
70
Several years ago, I noticed that there was only one significant variable in all of the nutritional “studies” I could find.
“Ioannidis wants rigor – full registration before the study, full transparency afterwards, fewer studies over all, but with better design.”
Good luck with that. As I pointed out in my article http://nequals1health.com/significant-variable-nutritional-study/ (written about 3 years ago), those restrictions are laughably easy to get around. All you have to do is run some secret research that enables you to tweak your inputs to get whatever result you need for your next “research” grant prior to doing the actual “study.” That’s why you see “studies” that compare “whole grain” with “processed grain,” but you will *NEVER* see a study that compares grain to NO grain at all.
BTW, I think 85% is a low estimate. Roughly 15% low.
10
And, thanks to What Really Happened,
http://www.whatreallyhappened.com
I came across this gem.
“A bug in fMRI software could invalidate 15 years of brain research”
http://www.sciencealert.com/a-bug-in-fmri-software-could-invalidate-decades-of-brain-research-scientists-discover
20
Interesting site:
It currently advises:
10
Just one issue – what does “detrended” mean?
10
Do you think drug and alcohol counsellors needed research to tell them that drug users tend to use more drugs when their pay/ pension arrives ? Or that if their clients spend most of their time with other active drug users they are more likely to use drugs ? Well if they did need to know then there is funded research available to “prove” it !
00
I’ve said it many times before – there are so many parallels between funded medical and funded climate change research. Sad to say as I’m in the healthcare business.
Both elements are questionable at best and more correctly outright fraudulent. But in the name of progress………..
Once you start playing with data it’s all over.
00