All randomized controlled trials aren't good and all observational trials aren't bad. Whew. It feels good to get that information out there.
Too many heuristics floating around to do any body any good. Lets take a sensible approach to evaluating the quality of evidence.
As journalists, writers, and healthcare stakeholders it is important to not grab that heavy peach. Low-hanging fruit is the manifestation of simplistic rules of thumb--RCTs are the standard for which we measure all clinical evidence for example.
First, a little context to the discussion of evidence-based medicine. One of the main themes of the Lown Annual Conference, is a must read.
Too many heuristics floating around to do any body any good. Lets take a sensible approach to evaluating the quality of evidence.
As journalists, writers, and healthcare stakeholders it is important to not grab that heavy peach. Low-hanging fruit is the manifestation of simplistic rules of thumb--RCTs are the standard for which we measure all clinical evidence for example.
First, a little context to the discussion of evidence-based medicine. One of the main themes of the Lown Annual Conference, is a must read.
As Evidence-based medicine (EBM) became more influential, it was also hijacked to serve agendas different from what it originally aimed for. Influential randomized trials are largely done by and for the benefit of the industry.
Meta-analyses and guidelines have become a factory, mostly also serving vested interests. National and federal research funds are funneled almost exclusively to research with little relevance to health outcomes. We have supported the growth of principal investigators who excel primarily as managers absorbing more money.
Diagnosis and prognosis research and efforts to individualize treatment have fueled recurrent spurious promises. Risk factor epidemiology has excelled in salami-sliced data-dredged articles with gift authorship and has become adept to dictating policy from spurious evidence. Under market pressure, clinical medicine has been transformed to finance-based medicine.
In many places, medicine and health care are wasting societal resources and becoming a threat to human well-being. Science denialism and quacks are also flourishing and leading more people astray in their life choices, including health. EBM still remains an unmet goal, worthy to be attained.-- Evidence-based medicine has been hijacked: a report to David Sackett
High Quality of the Evidence for Medical and Other Health-Related Interventions was Uncommon in Cochrane Systematic Reviews reports that of 1394 systematic reviews published on the Cochrane Database from January 2013 to June 2014, only 608 (43.6%) incorporated Grades of Recommendation, Assessment, Development, and Evaluation (GRADE). You may also be surprised by the quality of evidence reported for the first listed primary outcome:
- High quality 13.5%
- Moderate quality 30.8%
- Low quality 31.7%
- Very low level quality 24%
•Consideration of all outcomes--only 19.1% had at least one outcome with high quality of evidence.
•Reviews with high quality of evidence, only 25 had both significant results and a favorable interpretation of the intervention.
Dr. John Ioannidis also discussed findings of The Life Cycle of Translational Research for Medical Interventions providing an insightful graphic to demonstrate the problematic task of drug discovery. What you can observe shows the "median translation lag was 24 years (interquartile range, 14 to 44 years) between first description and earliest highly cited article. This was longer on average (median 44 versus 17 years) for those interventions that were fully or partially "refuted" (contradicted or having initially stronger effects) than for nonrefuted ones (replicated or remaining unchallenged) (P = 0.004)."
John Ioannidis, MD shared with us the following from Head-to-head randomized trials are mostly industry sponsored and almost always favor the industry-sponsor:
•Among trials published in 2011, 55/57 of non-inferiority trials with head to head comparisons sponsored by the industry demonstrated non-inferiority
•Success rate 96.5%
Flacco et al, J Clin Epidemiol 2015
Legend from article. The figure below describes the network of cosponsorship for the 12 most prolific companies.
Each company is shown by a node whose diameter is proportional to the number of trials sponsored.
Lines represent cosponsorship between companies, with thickness proportional to the number of trials cosponsored.
Dashed lines refer to comparison with only one cosponsored trial. The thickness of the autoloops is proportional to the number of trials where the respective company is the unique sponsor.
Red circles into each node represent the proportion of trials sponsored (or cosponsored) by each company that showed favorable results to the specific company (corresponding percentages are listed next to each node). BMS, Bristol-Myers Squibb.
Each company is shown by a node whose diameter is proportional to the number of trials sponsored.
Lines represent cosponsorship between companies, with thickness proportional to the number of trials cosponsored.
Dashed lines refer to comparison with only one cosponsored trial. The thickness of the autoloops is proportional to the number of trials where the respective company is the unique sponsor.
Red circles into each node represent the proportion of trials sponsored (or cosponsored) by each company that showed favorable results to the specific company (corresponding percentages are listed next to each node). BMS, Bristol-Myers Squibb.
 |  |
I will continue to post insights and steps we can employ to carefully analyze the quality of evidence behind the clinical research and outcomes reported in the literature. My point is to be aware of heuristics that suggest "always" or "never". You can see that quality is deeper than just the design of the study.
If upstream is dirty, downstream will be muddy – Tibetan Proverb
If upstream is dirty, downstream will be muddy – Tibetan Proverb
