The graphs are released on Tuesdays, facilitated on Wednesdays and discussed with statistical follow-up on Fridays--all with support of the American Statistical Association.
Here is the beauty of the method:
- What do you notice? I usually warm up the audience with an interesting piece of art. Showing audiences how their perception and bias can influence what they see is a great eye opener.
- What do you wonder? What is out of the frame or what do you assume is outside the frame? Is there data missing from the graphic that is needed for clarity?
- What might be going on in this graph? This is perfect for journalists. Catchy headlines are one thing but what in the graphic actually supports this claim?
Bring experts up on the stage with you. Clearly it isn't feasible to grab the actual scientist/physician, or expert but when you familiarize yourself with their body of work you can integrate their insights into your discussions. Plenary Sessions by Vinay Prasad MD, MPH is an essential part of my podcast habit especially since I work with a lot of oncology data.
You can easily apply the principles of the NY Times, What's Going on in This Graph lessons to your understanding of clinical publications. The recent JAMA Oncology journal article, Profiling Preexisting Antibodies in Patients Treated With Anti–PD-1 Therapy for Advanced Non–Small Cell Lung Cancer introduced a familiar bias as identified by Vinay Prasad and others, the guaranteed-time bias or how I recall learning about it--immortal-time bias.
In a nutshell, this means "immortal time is a span of cohort follow-up during which, because of exposure definition, the outcome under study could not occur". This can be by trial design, death, or the fact that the study outcome can not occur--because perhaps the study participant did not live long enough or dropped out of the study.
So what do you notice?
The objective of the JAMA Oncology article states the following, "To assess the safety and efficacy of anti–PD-1 treatment in patients with subclinical disease with advanced NSCLC and with or without preexisting autoimmune markers, including rheumatoid factor, antinuclear antibody, antithyroglobulin, and antithyroid peroxidase; and to assess potential clinical biomarkers that may be meaningfully and conveniently associated with clinical benefit or with irAEs following anti–PD-1 treatment."
I notice that you have to search until figure 3 to notice that there is no statistical difference in overall survival between patients with or without pre-existing autoimmune markers (rheumatoid factor, antinuclear antibody, antithyroglobulin, and antithyroid peroxidase).
I also noticed that Main Outcomes and Measures section ignore overall survival and only state the measures such as progression free survival (PFS) and immune-related adverse events (irAEs), which by the way, did report p-values < 0.05.
The antibody markers are either present at baseline or not. All good here. A hazard ratio of 0.72 with a p value of 0.19 are arguably nothing to write home about.
Now you might have missed this without being guided by the discussion in the Plenary Sessions podcast but the paper does away with the overall survival argument until we view graphs comparing irAEs. Why does this matter? Unlike the presence of antibodies at or before time "0", adverse events occur after time "0". You have to receive the treatment before you can experience an adverse reaction.
This is where the immortal-time bias becomes relevant.
What do you need in order to experience adverse events? You need to be alive. These irAEs have been associated with response to nivolumab. Perhaps the group with irAEs have longer overall survival because they had to survive to have the immune related adverse event.
See the problem here?
The potential for guarantee-time bias (GTB), also known as immortal time bias, exists whenever an analysis that is timed from enrollment or random assignment, such as disease-free or overall survival, is compared across groups defined by a classifying event occurring sometime during follow-up.
The types of events associated with GTB are varied and may include the occurrence of objective disease response, onset of toxicity, or seroconversion. However, comparative analyses using these types of events as predictors are different from analyses using baseline characteristics that are specified completely before the occurrence of any outcome event.--Challenges of guarantee-time bias. Giobbie-Hurder , Gelber RD, Regan MM.
There are ways (likely known to the authors) to avoid this type of bias. For starters, use landmark survival analysis to plot all patients alive at say week 10. Did the patients with immune-related adverse events have better outcomes? Read this article if you are interested in details of avoiding time bias in clinical trials, Challenges of guarantee-time bias. Here is a visually helpful description from the research paper.
Patients who undergo transplantation seem to have longer survival times. However, patients in the transplantation group must survive at least until donor is available (blue), and their survival reflects total time before and after transplantation (blue and gold).
More favorable survival does not result from the effect of transplantation but rather from the inclusion of extra waiting time (guarantee time). By contrast, the no-transplantation group includes all patients who died before a suitable donor could be found.
If a picture is worth a thousand words, a graphic is at least worth a million numbers.