All randomized controlled trials aren't good and all observational trials aren't bad. Whew. It feels good to get that information out there.
Too many heuristics floating around to do any body any good. Lets take a sensible approach to evaluating the quality of evidence.
As journalists, writers, and healthcare stakeholders it is important to not grab that heavy peach. Low-hanging fruit is the manifestation of simplistic rules of thumb--RCTs are the standard for which we measure all clinical evidence for example.
First, a little context to the discussion of evidence-based medicine. One of the main themes of the Lown Annual Conference, is a must read.
As Evidence-based medicine (EBM) became more influential, it was also hijacked to serve agendas different from what it originally aimed for. Influential randomized trials are largely done by and for the benefit of the industry.
High Quality of the Evidence for Medical and Other Health-Related Interventions was Uncommon in Cochrane Systematic Reviews reports that of 1394 systematic reviews published on the Cochrane Database from January 2013 to June 2014, only 608 (43.6%) incorporated Grades of Recommendation, Assessment, Development, and Evaluation (GRADE). You may also be surprised by the quality of evidence reported for the first listed primary outcome:
•Consideration of all outcomes--only 19.1% had at least one outcome with high quality of evidence.
•Reviews with high quality of evidence, only 25 had both significant results and a favorable interpretation of the intervention.
Dr. John Ioannidis also discussed findings of The Life Cycle of Translational Research for Medical Interventions providing an insightful graphic to demonstrate the problematic task of drug discovery. What you can observe shows the "median translation lag was 24 years (interquartile range, 14 to 44 years) between first description and earliest highly cited article. This was longer on average (median 44 versus 17 years) for those interventions that were fully or partially "refuted" (contradicted or having initially stronger effects) than for nonrefuted ones (replicated or remaining unchallenged) (P = 0.004)."
John Ioannidis, MD shared with us the following from Head-to-head randomized trials are mostly industry sponsored and almost always favor the industry-sponsor:
•Among trials published in 2011, 55/57 of non-inferiority trials with head to head comparisons sponsored by the industry demonstrated non-inferiority
•Success rate 96.5%
Flacco et al, J Clin Epidemiol 2015
Legend from article. The figure below describes the network of cosponsorship for the 12 most prolific companies.
Each company is shown by a node whose diameter is proportional to the number of trials sponsored.
Lines represent cosponsorship between companies, with thickness proportional to the number of trials cosponsored.
Dashed lines refer to comparison with only one cosponsored trial. The thickness of the autoloops is proportional to the number of trials where the respective company is the unique sponsor.
Red circles into each node represent the proportion of trials sponsored (or cosponsored) by each company that showed favorable results to the specific company (corresponding percentages are listed next to each node). BMS, Bristol-Myers Squibb.
I will continue to post insights and steps we can employ to carefully analyze the quality of evidence behind the clinical research and outcomes reported in the literature. My point is to be aware of heuristics that suggest "always" or "never". You can see that quality is deeper than just the design of the study.
If upstream is dirty, downstream will be muddy – Tibetan Proverb
Sign up for our newsletter!
Browse the archive...
Thank you for making a donution!
In a world of "evidence-based" medicine I am a bigger fan of practice-based evidence.
Remember the quote by Upton Sinclair...
“It is difficult to get a man to understand something, when his salary depends upon his not understanding it!”