Numeracy is a big hurdle in the sciences. Listening to a talking head blah, blah, blah, about statistics isn't going to enlighten anyone or decrease the learning curve. I think we lost something from the Journal Club days...a good interactive conversation about data.
Case in point. Industry migrated toward real world evidence without an infrastructure to support the knowledge shift. Experts in the field commented at DIA/FDA Statistics Forum, "We weren't trained how to model the emerging disparate data sources".
Only the data isn't waiting--pragmatic trials are evolving into a valuable tool to examine real world evidence.
In the pragmatic-explanatory continuum, a randomized controlled trial (RCT) can at one extreme investigate whether a treatment could work in ideal circumstances (explanatory), or at the other extreme, whether it would work in everyday practice (pragmatic). How explanatory or pragmatic a study is can have implications for clinicians, policy makers, patients, researchers, funding bodies, and the public.--Tosh and colleagues 2011
Maybe it is time to utter the dirty little words that are spawning business models faster than you can say checkpoint inhibitor--claims data. A recent article published in JAMA Cardiology does a nice job but I think it might be useful to loop back on some of the dialogue--journal club style.
Accuracy of Medical Claims for Identifying Cardiovascular and Bleeding Events After Myocardial Infarction A Secondary Analysis of the TRANSLATE-ACS Study
I suggest reading the full paper at the link. The point of sharing the article with a 30,000 foot view is to hopefully level-set conversations around the magical world of claims data. And a brief overview of what we should glean from the insightful work of the authors.
In short, the authors performed a post-hoc analysis of the TRANSLATE-ACS observational study to compare incidences of bill-identified events by either medical claims data or by physician adjudication to identify the accuracy in identifying potential outcomes.
The k statistic is a measure of inter-rater agreement (between the physician adjudicator and medical claims data). They are classifying the patients or events into "mutually exclusive" events.
Data were analyzed from January 30, 2015, to March 2, 2017. We calculated the total number of each event type when identified by medical claims vs when physician adjudicated. We also calculated cumulative incidence rates at the patient level of each event type and the combined outcome of death, MI, and stroke when defined by the 2 respective methods.
Although somewhat arbitrary, kappa of 0-0.20 may be identified as slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1 as almost perfect. Others consider kappas > 0.75 as excellent, 0.40-0.75 as fair to good, and < 0.40 as poor.
Event rates at 1 year were lower for MI, stroke, and bleeding when medical claims were used rather than physician adjudication. Moderate agreement between medical claims and physician adjudication was observed in ascertaining MI and stroke events, but agreement was worse for bleeding events. While medical claims may be a reasonable resource to assess MI and stroke outcomes, caution is still needed. Medical claims have limited accuracy in identifying bleeding events, which suggests the need for an alternative approach to ensure good safety surveillance in cardiovascular studies.
Summarized by FDA/DIA statistics NIH Collaboratory: