Navigating High-Value Care

A graph is worth a million numbers...

1/5/2019

I learned and promptly incorporated this creative method of looking at graphs and deciphering their meaning for a section of the data literacy courses I teach.

The graphs are released on Tuesdays, facilitated on Wednesdays and discussed with statistical follow-up on Fridays--all with support of the American Statistical Association.

Here is the beauty of the method:

What do you notice? I usually warm up the audience with an interesting piece of art. Showing audiences how their perception and bias can influence what they see is a great eye opener.
What do you wonder? What is out of the frame or what do you assume is outside the frame? Is there data missing from the graphic that is needed for clarity?
What might be going on in this graph? This is perfect for journalists. Catchy headlines are one thing but what in the graphic actually supports this claim?

An important habit to integrate into any presentation or workshop--or even anything you write, is the following.

Bring experts up on the stage with you. Clearly it isn't feasible to grab the actual scientist/physician, or expert but when you familiarize yourself with their body of work you can integrate their insights into your discussions. Plenary Sessions by Vinay Prasad MD, MPH is an essential part of my podcast habit especially since I work with a lot of oncology data.

You can easily apply the principles of the NY Times, What's Going on in This Graph lessons to your understanding of clinical publications. The recent JAMA Oncology journal article, Profiling Preexisting Antibodies in Patients Treated With Anti–PD-1 Therapy for Advanced Non–Small Cell Lung Cancer introduced a familiar bias as identified by Vinay Prasad and others, the guaranteed-time bias or how I recall learning about it--immortal-time bias.

In a nutshell, this means "immortal time is a span of cohort follow-up during which, because of exposure definition, the outcome under study could not occur". This can be by trial design, death, or the fact that the study outcome can not occur--because perhaps the study participant did not live long enough or dropped out of the study.

So what do you notice?

The objective of the JAMA Oncology article states the following, "To assess the safety and efficacy of anti–PD-1 treatment in patients with subclinical disease with advanced NSCLC and with or without preexisting autoimmune markers, including rheumatoid factor, antinuclear antibody, antithyroglobulin, and antithyroid peroxidase; and to assess potential clinical biomarkers that may be meaningfully and conveniently associated with clinical benefit or with irAEs following anti–PD-1 treatment."

I notice that you have to search until figure 3 to notice that there is no statistical difference in overall survival between patients with or without pre-existing autoimmune markers (rheumatoid factor, antinuclear antibody, antithyroglobulin, and antithyroid peroxidase).

I also noticed that Main Outcomes and Measures section ignore overall survival and only state the measures such as progression free survival (PFS) and immune-related adverse events (irAEs), which by the way, did report p-values < 0.05.

The antibody markers are either present at baseline or not. All good here. A hazard ratio of 0.72 with a p value of 0.19 are arguably nothing to write home about.
Figure 3

What do you wonder?

Now you might have missed this without being guided by the discussion in the Plenary Sessions podcast but the paper does away with the overall survival argument until we view graphs comparing irAEs. Why does this matter? Unlike the presence of antibodies at or before time "0", adverse events occur after time "0". You have to receive the treatment before you can experience an adverse reaction.

This is where the immortal-time bias becomes relevant.

What do you need in order to experience adverse events? You need to be alive. These irAEs have been associated with response to nivolumab. Perhaps the group with irAEs have longer overall survival because they had to survive to have the immune related adverse event.

See the problem here?

The potential for guarantee-time bias (GTB), also known as immortal time bias, exists whenever an analysis that is timed from enrollment or random assignment, such as disease-free or overall survival, is compared across groups defined by a classifying event occurring sometime during follow-up.

The types of events associated with GTB are varied and may include the occurrence of objective disease response, onset of toxicity, or seroconversion. However, comparative analyses using these types of events as predictors are different from analyses using baseline characteristics that are specified completely before the occurrence of any outcome event.--Challenges of guarantee-time bias. Giobbie-Hurder , Gelber RD, Regan MM.

Figure 1 cleverly moved to the front of the article regardless of its dubious immortal time-bias and surrogate end-point.

Figure 2 below is looking at progression-free survival only with or without presence of any antibody

There are ways (likely known to the authors) to avoid this type of bias. For starters, use landmark survival analysis to plot all patients alive at say week 10. Did the patients with immune-related adverse events have better outcomes? Read this article if you are interested in details of avoiding time bias in clinical trials, Challenges of guarantee-time bias. Here is a visually helpful description from the research paper.

Patients who undergo transplantation seem to have longer survival times. However, patients in the transplantation group must survive at least until donor is available (blue), and their survival reflects total time before and after transplantation (blue and gold).

More favorable survival does not result from the effect of transplantation but rather from the inclusion of extra waiting time (guarantee time). By contrast, the no-transplantation group includes all patients who died before a suitable donor could be found.

Have data questions? Send them along...

If a picture is worth a thousand words, a graphic is at least worth a million numbers.

The curse of bigness--revisited again and again

12/16/2018

If you work in health policy, medicine, or health economics I recommend you listen to The Week in Health Law podcast. I remove pods from my playlist all the time. Occasionally from a perceived terse exchange on twitter about a subscription paywall but more often influenced by verbal tics such as vocal fry and upspeak, or limited perspectives on complex topics.

You don't get those distractions when listening to discussions of health policy law on the TWIHL podcast. North Carolina is finally sweating off the snow from a record snowfall--that I dodged during travels to DC and NYC. I laced up my sneakers and headed out the door for a misty run along the creek. The podcast queue always keeps me company.

This whole brain thread was launched by a simple question I was asked on social media. How do I use cost information (data) for gap analysis in medical education. There are too many data points to answer the question simply.

Listening to a technical podcast by experts in health law creates opportunities to dig a little deeper into questions. And to question long-held "answers" roaming around the status quo. To appreciate the impact of legal precedent, policy, and a new digital economy still regulated or "guiding" by outdated regulations enacted before we had our current technologic advances--we need context.

When should "guidance" yield to regulation? When do we reach the point of "too late", the toothpaste is already out of the tube?

I suggest that data questions need to be informed by the realities of choice at the point of care.

Who is really making the decisions that influence patient care?

A common theme throughout is "bigness." How big should companies, industries, governments be allowed to get before the ill and often unintended consequences are all but palpable? Not only what is the role of regulation but is there historical precedence? Is there something we can learn?

The Curse of Bigness by Christopher Ketcham

THE NEXT TIME I HEAR a politico or banker or Detroit executive talk about institutions “too big to fail,” I’ll direct them to the 34 percent of Americans who are obese.

Last I heard, these big Americans, themselves a kind of cultural institution, were failing en masse, racked by diabetes, asthma, heart trouble, and bound for early death. The human form can only grow so big. Or I could point them to Pig #6707.

Conceived in the laboratories of the U.S. Department of Agriculture during the 1990s, Pig #6707’s embryo was genetically altered with a human growth gene to develop a super-pig, bigger and faster-growing and more productive of meat.

But the genetic alterations produced a monster, impotent and nearly blind, its legs arthritic, its body crippled, the creature able to stand up and be photographed only with the support of a plywood board.--Christopher Ketcham

Today's episode featured discussion of a roundtable discussion on "The Law and Policy of AI, Robotics, and Telemedicine in Health Care." I use SIRI on my phone to create notes so that when I return to my office, I can recall the vast number of rabbit holes I will enjoy rappelling down as time permits. What follows is a summary of a few insights with links so you can also design an exploration based on your interests but first, a little back story.

I don't know about you, but I have not heard the tech industry discuss publicly the concerns of bias and discrimination integrated into AI or other devices. The solution to our healthcare woes are just a click away as algorithms replace human cognition and we seek hacks to fast track our journey to better health.

For starters, we lack a robust definition of device that differs in an appreciable way from that of obscenity, "I know it when I see it." -- 1964 United States Supreme Court Justice Potter Stewart.

The Lochnerized first amendment and the FDA: toward a more democratic political economy

Lochnerizing refers to a ruling in a 1905 Supreme Court case where corporations were granted constitutional rights by invalidating democratically enacted laws. For example, the case struck down a New York State law limiting the number of hours a baker could work in one day citing the law would violate the constitutional liberty of a contract, between two "persons."

The Columbia Law Review published this informative article as a response to the Columbia Law Review's 2018 Symposium. The article discusses in a non-trivial manner the weaponizing of the first amendment and the impact it has had, and will have on the Food and Drug Administration (FDA).

“[T]he majority has chosen the winners by turning the First Amendment into a sword, and using it against workaday economic and regulatory policy. Today is not the first time the Court has wielded the First Amendment in such an aggressive way. And it threatens not to be the last. Speech is everywhere—a part of every human activity (employment, health care, securities trading, you name it). For that reason, almost all economic and regulatory policy affects or touches speech. So the majority’s road runs long. And at every stop are black-robed rulers overriding citizens’ choices. The First Amendment was meant for better things.”--Janus v. AFSCME, Justice Kagan dissenting

It is an important discussion on a current trend of the U.S. courts to "pivot away from the model of private market regulation upon which the FDA is approach is built."

In this era of Too Big To Fail we could do worse than revisit the writings of Louis D. Brandeis, Supreme Court judge from 1916 to 1939 and prolific author. Here is a taste of his views on concentration of power within banking industry, What Publicity Can Do.

THE NEW UTILITIES: PRIVATE POWER, SOCIAL INFRASTRUCTURE, AND THE REVIVAL OF THE PUBLIC UTILITY CONCEPT

This final article is an important one as it reminds us of the earlier landscape, reviews the current challenges, and puts forward potential solutions. There is a tendency to silo much of the digital debates around privacy and the rights of consumers without rolling the healthcare wagon into the debate.

From the control of banks over financial stability as well as access to finance and credit, to the control of internet service providers (ISPs) and telecom companies over broadband infrastructure, the problem is the same: private actors possess the means to undermine the public value of essential goods and services upon which many businesses, communities, and individuals depend.--THE NEW UTILITIES: PRIVATE POWER, SOCIAL INFRASTRUCTURE, AND THE REVIVAL OF THE PUBLIC UTILITY CONCEPT--K. Sabeel Rahman

There is a powerful delineation between power brokers in healthcare and consumers rebranded as patients, wielding market power now empowered as patient-centeredness--heralded as a way to equalize the debate.

Definition of social infrastructure--” Where private actors accumulate outsized control over those goods and services that form the vital foundation or backbone of our political economy—social infrastructure—this control poses dangers. By defining social infrastructure as a concept, this article provides a way to diagnose essential goods and services and therefore potentially problematic accumulations of private power."--THE NEW UTILITIES: PRIVATE POWER, SOCIAL INFRASTRUCTURE, AND THE REVIVAL OF THE PUBLIC UTILITY CONCEPT--K. Sabeel Rahman

Collectively we praise industries such as pharmaceuticals and healthcare technology when they "do good" through public endowments, investments, and charitable giving but we rarely if ever challenge them to do less harm through persistent self-serving policies that limit access to medications and prolong financial benefits. The threat is always around innovation.

Drawing from these disparate debates over net neutrality and TBTF financial firms, I extract four key elements of a twenty-first century framework for public utility regulation: firewalling core necessities away from behaviors and practices that might contaminate the basic provision of these goods and services—including through structural limits on the corporate organization and form of firms that provide infrastructural goods; imposing public obligations on infrastructural firms, whether negative obligations to prevent discrimination or unfair disparities in prices or positive obligations to proactively provide equal, affordable, and accessible services to under-served constituencies; and creating public options, state-chartered, cheaper, basic versions of these services that would offer an alternative to exploitative private control in markets otherwise immune to competitive pressures.--THE NEW UTILITIES: PRIVATE POWER, SOCIAL INFRASTRUCTURE, AND THE REVIVAL OF THE PUBLIC UTILITY CONCEPT--K. Sabeel Rahman

The Aspen Institute also discussed the liabilities of bigness and what we can and can't accomplish.

These final quotes from the article speak to the rise of social determinants and constructs into the debate surrounding reform. I mean true reform. Not the current model of working within a broken system as we tiptoe around real change.

Progressive reformers thus understood public utilities not just in terms of economies of scale and laws of nature, but in moral and social terms. Industries triggered public utility regulation when there was a combination of economies of scale limiting ordinary accountability through market competition and a moral or social importance that made the industries too vital to be left to the whims of the market or the control of a handful of private actors.

This combination of economic dominance and social necessity is what created the threat of not just exploitative prices but also discrimination and unequal access. The public utility model offered a way to check this particular form of private power. Public utility regulation thus did not apply to all forms of private power—but it did address the most vexing, powerful, and dangerous private actors. At the same time, this emphasis on social importance meant that industries could evolve into utilities over time, through changes in their production and distribution model and through social changes whereby one or another good became essential to more people. THE NEW UTILITIES: PRIVATE POWER, SOCIAL INFRASTRUCTURE, AND THE REVIVAL OF THE PUBLIC UTILITY CONCEPT--K. Sabeel Rahman

These are thoughtful discussions that deserve consideration as we struggle with enormous inequality and unsurmountable financial burdens in society as a whole.

...firms and sectors that might warrant greater regulatory oversight by examining three overlapping conditions: the economics of production; the downstream uses of the good or service; and the degree to which the good or service is a necessity that makes its users particularly vulnerable to exploitation. The presence of all three features indicates a firm or sector that is “infrastructural,” where the concentration of private power over these services poses a unique potential threat to public welfare. THE NEW UTILITIES: PRIVATE POWER, SOCIAL INFRASTRUCTURE, AND THE REVIVAL OF THE PUBLIC UTILITY CONCEPT--K. Sabeel Rahman

Be an outlier--the view is better

12/12/2018

My favorite recent article about journalists was sent to me by a colleague. An article over on Vox.com titled, My advice for aspiring explainer journalists. There is a lot of good information nestled within and more than a few whiffs of irony.

Explaining that most journalists (employed) are white, elite, and went to an exclusive college while sprinkling words like "epistemic crisis" and inserting logarithmic curves to explain distribution of knowledge amused me. But David Roberts also candidly reveals the obvious secret we all know but don't talk about--you don't need journalism school--gasp.

Our new digital landscape has created fenceless terrain where we can hopefully step outside of google analytics and distill granular elements of stories, contextualized to answer the "so what" metric.

Because they were supply constrained, newspapers and newspaper journalists focused on what was new, what just happened, the incremental development.

But lots of times, readers had no way of making sense of those developments or contextualizing them. They were getting the leaves, but they’d never gotten the trunk.--David Roberts

I never fancied myself an "explainer" journalist but here we are. I like to think a bit wider about an issue and hopefully start meaningful conversations. When I go to conferences I never know where the flare will be found. Many colleagues report the facts and nothing but the facts but how do you know they are indeed factual?

Ahhhh...have we stumbled upon an epistemic crisis?

I recently spoke to a large group of Tableau users in Raleigh. I called the talk "We Can Only Connect the Data we Collect" to remind those of us in the data and analytics space to be mindful of data sourcing.

Are we using data because it is easy to find or because it answers the carefully formulated question we designed?

The new year is full of interesting developments.

I will be finishing up the Applied Data Science certificate from Columbia Fu Foundation school of Engineering and Applied Science.
Leading more workshops in Data Literacy--you can request more info here
Publishing a book, Autobiographicacy: a year visualizing data
Releasing the new updated newsletter. We went on hiatus when we reached our new subscription level. Happy to bring immediately useable information and relevance to our heterogenous subscribers
Creating a bigger presence on Tableau Public to share interactive visualizations

thank you for your support...

I always appreciate the support of donors and sustaining subscribers.

Here are a few perks of contributing
(in addition to no adds):

data visualizations to use in proposals or grants (just ask)
review of outcomes or data models
A free 15 minute consult
Pre-release copy of book, Autobiographicacy: a year visualizing data

extracting essential data elements--one drop at a time.

12/6/2018

I irregularly visit NYC on a somewhat regular basis. Business meetings, visits with family, museums--and wonderful friends where we pick up where we left off.

Kathyann and I hunted for a place to grab some lunch and talk about what old friends talk about. We settled on Distilled NY and the menu graphic captured my attention.

I realized I am basically a data distillery.

It doesn't stop there. When you work in a field where you are often one of only a handful of women invited into conversations you see things. I seek out colleagues with design aesthetics, indomitable spirits, and perseverance toward an outcome or goal. When you can call them friend--all the better.

The aforementioned Kathyann designs jewelry that matters. There is a Power Collection that reminds me of the boom and pow of old batman cartoons.

Near and dear to my heart is the I AM COLLECTION. I never had the right answer when confronted with the question, "what are you?" growing up biracial during the 70s.

Kathyann taught me the answer--anything I want to be.

How does she describe her designs? "Uncomplicated, imbued with meaning and intention"

Amy Herman taught me a new way to think about data. I was intrigued by her work teaching visual intelligence at the Frick and connected with her book Visual Intelligence: Sharpen Your Perception, Change Your Life.

I have always had a passion for art. Traveling for work I prioritized visits to local museums over cocktail socials with clients or late evenings in hotel restaurants. The lessons I have learned help me engage audiences with exercises in perception and bias before we even look at a chart or a graphic.

My collaborators enjoy the sketches and design thinking pulled into creative briefs to better visualize scope and design of data projects big and small. You have Amy to thank.

Watch the freshly minted TED talk below. Distill it down to the 4 "A"s as described in the TED talk, ASSESS, ANALYZE, ARTICULATE, and ACT--and wait for the donut. There is always a donut.

You pretty much can’t get away from bacon or whiskey in the south. Put a doughnut in it and you’d be good to go—Hillary Scott, American country music singer-songwriter

Your data is a mess...

12/2/2018

I primarily use data sources that are publicly available. I travel around teaching data literacy and want to share data anyone can use to answer their data questions or hone business insights into actionable tactics. My biggest pet peeve is being sent a tool to simplify data curation--and then a bill several weeks later. I like a challenge so I encourage colleagues to learn a few skills to eliminate the middle man.

Look at this ugly little packed bubble chart. They are horrible little charts. Our eyes are not able to differentiate quantities in spherical formats. Okay maybe some do look bigger but by how much? I picked this monstrosity because while searching through clinicaltrials.gov for recruiting trials sorted geographically I noticed how a lack of standardization during data collection can leave chaos.

Another obvious data flaw is the wide variety of labels for studies within the database. What influences the classification scheme by colorectal cancer, colon cancer, colorectal carcinoma or even colorectal adenocarcinoma?

Tableau Prep is useful for sorting through the mess with a wide variety of tools to help clean up a dataset. I will admit I can be a little loosey goosey when analyzing in discovery mode. This particular task will be shared with an audience hoping to improve their skills at utilizing clinicaltrials.gov database. Look at the lack of clarity in how the conditions are captured...

How can you be certain you are capturing data about clinical trials of patients with colorectal cancer when there are so many different descriptions of similar but perhaps not identical characterizations?

Following a bit of data wrangling I was able to filter down to 3 broad categories shown to the right of the original snapshot. I keep them grouped so I can still drill down if I need more granularity or want to verify the findings. This visualization below is not with the clean version of grouping conditions--I was actually looking for specific types of Phase II data. This included editing classifications captured as Phase 1/2, Phase 2, Phase I/II, and Phase II...yikes.

What I am actually doing is comparing demographics between clinical trial participants and Phase II historical proprietary research data I have permission to access. How have the populations included or excluded from clinical trials in colorectal cancer changed, evolved, or stayed the same?

Can you see why this might be relevant in this current healthcare ecosystem where the prices of oncology drugs continue to escalate but the clinical efficacy is ill-defined?

Progression-free survival is a recurrent primary endpoint used to rate and compare randomized clinical trial findings but what does this mean for point of care decisions between community oncologists and their patients with colorectal cancer?

"We can only connect the dots we collect"...Amanda Palmer

Selling pain repackaged as freedom...

11/19/2018

Healthcare is a system many of us seek to change. But what might actually be happening? We are slowly being changed by the system. What if we succeeded in improving health across our nation? Hospitals would close, economies would shift, and profits would diminish.

What happens if we reimagine patent law and large profitable pharmaceutical companies are challenged to introduce generic and more affordable drugs onto the market much sooner?

Introducing a little transparency into the actual costs borne by industry would allow better solutions to be developed and imagined. We shouldn't ignore the research & development in academic settings and how that is funded by tax-payer dollars. No. Not in every case but we aren't allowed to fill silences with perturbations of a system where so few profit at the expense of all of us.

What if we stop believing that innovation is fueled by economic excess instead of economies of sustainability, technology, and viability?

Have you ever heard an industry client question the profits they are making on their blockbuster drugs or only lamenting that they deserve the windfall because of other pipelines that were silently halted?

And I basically became very interested in the silences, what we were not allowed to talk about or what we, just by custom, didn’t talk about when we came together to talk about making the world better.

So the Aspen consensus was: You can tell the rich and powerful in our age to do more good, but you can never tell them to do less harm. You can tell them to give more, but you can’t tell them to take less. You can tell them to share the spoils of extreme capitalism, but you can’t tell them to renovate capitalism.--Anand Giridharadas

Read the latest headlines about Michael Bloomberg's ginormous investment in the Johns Hopkins? Anand also asks why not question a system where billionaires have so much excess to donate?

Working at the level of systems means that problem components are in interdependent and constantly changing relationships. Linear approaches that address one component or factor at a time are inadequate in addressing dynamic conditions.

Stakeholders show a variety of motives and behaviors for engaging with systems. Designers must reconcile competing goals and performance expectations through participatory methods that value stakeholder input and ongoing feedback from the system.

And issues of sustainability, technological feasibility, and economic viability are crucial to making design solutions work over time.--Trend I: Complex Problems--Meredith Davis AIGA

I am working on a data literacy project where I am investigating the opioid crisis integrating data many have not considered. The question posed by Anand again is chilling and speaks to the larger driver of a crisis looking for linear solutions--that won't work.

Why write words into conversations if you are not willing to dig deeper and illuminate contradictions?

I think what the investigation has left me with is thinking about, even as a writer, how do I make sure that I am using my power, however limited it is, to interrogate systems and ask the questions you’re not supposed to be asking, instead of doing the kind of journalism or thinking or speaking that merely props up power.

But it also shows up for me every day in thinking about, am I using my voice to say inconvenient things that might cause me to lose friends or social capital, to put it in those market terms, but that are part of maybe pushing us, in some small way, to where we need to get as a society?--On Being Krista Tippett in conversation with Anand Giridharadas

Working as a consultant I was once jokingly accused of heckling the very client that hired me to provide expertise in immuno-oncology. I didn't see it that way but I was never brought in "live" again. The company, in my opinion, had made a bold choice in pre-clinical formulation regarding the FC region that triggers cell-mediated cytotoxic effector functions--in particular antibody dependent cellular cytotoxicity (ADCC).

I requested the pre-clinical data so I could review and see if there was perhaps a unique aspect of this particular molecule we could highlight to help differentiate a "not first to market" investigational drug. When I received the document, the section I requested was not included. I found this interesting but was dismissed and never saw the information. After a few conversations that included "mansplaining" from an apparent team Google search I could see the consensus irrevocably leaning toward the status quo--I dropped it but still recall how little we are willing to probe when trying to secure the next round of business.

All around, us the winners in our highly inequitable status quo declare themselves partisan of change. They know the problem, and they want to be part of the solution. Actually, they want to lead the search for solutions. They believe that their solutions deserve to be at the forefront of social change. They may join or support movements initiated by ordinary people looking to fix aspects of their society. More often, though, these elites start initiatives of their own, taking on social change as though it were just another stock in their portfolio or corporation to restructure. Because they are in charge of these attempts at social change, the attempts naturally reflect their biases.

The initiatives mostly aren't democratic, nor do they reflect collective problem-solving or universal solutions. Rather, They favor the use of the private sector, and its charitable spoils, the market way of looking at things, and the bypassing of government. They reflect a highly influential view that the winners of an unjust status quo--and the tools and mentalities and values that helped them win--are the secret to redressing the injustices. Those at greater risk of being resented in an age of inequality are thereby recast as our saviors from an age of inequality.--Winners Take All: the elite charade of changing the world--Anand Giridharadas

Try and find the book. I think it is an important discussion that needs to happen.

I agree that most of us are decent people trying to prop up an indecent system.

But whom are we serving when economics of billion dollar revenues are valued instead of improved patient outcomes, when investments are made in direct to consumer advertising, lobbyists, and the latest M&A are broadcast as successes?

There are harms associated with doing good. Why aren't we discussing them on a broader platform? Why are so many of us in pain? Is the solution a "pill" or upstream investments to reduce social vulnerabilities?

What are we actually being sold?

Your data team: "ontogeny recapitulates phylogeny"

11/12/2018

The dubious history of Ernst Haeckel and debunking of his biogenetic law aside, I still love the catchphrase he generated--"ontogeny recapitulates phylogeny." The embryo does in certain instances, reflect evolutionary history--pharyngeal gill slits anyone? But it isn't as reminiscent of Lamarckian ideology as our modern day understanding of the complexities of genetics reveal.

I recently heard the phrase attributed to nascent data teams. Think of the pluripotent stem cell for example. Many a data project begins with a small nimble team with an undifferentiated skill set. Angela Bassa, Director of Data Science at iRobot, shares that one of the earliest paths to "value" is by answering easy data queries with an architecture that takes care of itself.

Over time, a data team evolves and grows the ability to mature into anything. This leads to the specialization and differentiation of mature data teams. This sounds tremendously like how data projects evolve in the wild. When the journey is new, you may find yourself dabbling in a bit of the data engineering and data architecture bits regardless of focused expertise in formulating data questions.

The truth of the matter is, everyone has data. Data scientists worth their salt have the skill of formulating a data question and selecting the right statistical and visualization tools to formulate a solution. We strive for replicability, interpretability, and fit for purpose.

Full disclosure. When I am launching new data strategies with clients I focus on team building. One of the best pieces advice, I "borrowed" from Amazon CEO Jeff Bezos.

Perhaps you have heard of the 2 pizza rule.

"It's simple. The more people you pack into the meeting, the less productive the meeting will likely be. The solution?

Never have a meeting where two pizzas couldn't feed the entire group."--Business Insider

If you scour adds for data scientists, what you may notice is a focus on tools rather than asking if you have the imagination to envision what an answer to a data question might look like. In my opinion, the method should be second to the actual problem a team seeks to answer. It reminds me of a data professional that works within a large hospital system in my neighborhood. There are so many legacy systems in his department, they can barely communicate. The seasoned IT team has preferred ways of computing while the newer team members are more likely to be comfortable with the latest and greatest.

How many pizzas do you think it would take to feed those folks?

Listen to a recent post about managing data science teams--this post is a bit of a summary--the episode is worth your ears...

Secret ingredients of a robust medical education strategy

11/3/2018

Headlines within Continuing Medical Education broadcast record attendance at events, "Innovative" learning strategies, and tactics that lead us to believe we are making significant inroads in closing the gaps between evidence-based medicine and actual physician behavior at the point of care.

All those successes include big data integration and improved data literacy, or do they?

First, a little context. The CME industry includes close to $3 billion dollars in educational investments from a wide variety of financial categories including registration fees, commercial support, advertising and exhibits income, private donations, and government grants.

The ACCME's Accreditation Criteria require providers to produce educational activities that are designed to change competence, performance, or patient outcomes. Providers are then required to analyze the changes that were achieved as a result of the activities.
Figure 4 illustrates the percentage of CME provided in 2017 that was designed and/or analyzed for changes in competence, performance, and/or patient outcomes.--2017 ACCME Data Report

Here is where the rigor starts to unravel. The measurement of competence and performance is inherently subjective or observational and simultaneously not evaluated utilizing sophisticated analyses. And the teeny tiny attention to actual patient outcomes is alarming.

Likert metrics, poorly constructed surveys utilizing outdated research methodology, and narrow focus to measure the activity instead of the actual physician level engagement or impact on patient outcomes appears to go unnoticed.

The Accreditation Council for Continuing Medical Education (ACCME) President and CEO writes in the annual report:

CME has responded to the changing health environment. Healthcare is practiced in teams--and clinicians need to learn how to work effectively as teams for the benefit of their patients. While CME traditionally has been focused on physicians, it has never excluded other professions. This data report shows that the number of interactions with learners other than physicians, such as nurses, pharmacists, and physician assistants, has increased nearly 90% over the past 10 years. Overall, learner interactions (physicians and other learners) grew 57% during that time.

Why does this growth matter? Degree and training programs are the foundation of clinician education; for the rest of their careers, clinicians rely on accredited continuing education to help them provide high-quality, safe, compassionate, and effective care to the patients they serve. Those activities help clinicians absorb and process rapidly changing medical information and practice an ever-diversifying set of complex skills.--Graham McMahon, MD, MMSc Accreditation Council for Continuing Medical Education (ACCME)

Not to be all sturm and drang but I disagree regarding the claim that CME has responded to the changing health environment. I spend a lot of time in Washington DC and rarely see CME colleagues participating in discussions around value-based reform at the FDA, Brookings Institute, PCORI, Kaiser Family Foundation, or National Press Club.

Following a presentation at Alliance I was asked to write an article for their open access publication, Almanac. I decided to pull the draft when I was challenged to provide citations from publications to support the findings and assertions regarding the importance and relevance of System's Thinking in medical education. How can innovation occur if we continue to simply rinse and repeat what we have been doing for decades? Why are so few willing to be first--seed a hypothesis and proceed from there?

A recent small survey of ~100 writers of CME needs assessments should stimulate a larger query of hypothesis test pathways to improve how we engage and evaluate learning in continuing medical education. What it revealed--at least to me--is a potential skill desert unable to provide relevance and insight in our healthcare big data ecosystem.

I have offered over the last few years to help improve the research methodology but my offer never received a response. I also reach out to colleagues with insights revealed from global travel and experience with comparative health systems--also crickets.

Respondents of the annual survey were primarily staff employees 56% (n=60) followed by freelancers 33% (n=35). I hope next year they will ask about roles within the company and link it to educational levels or areas of study. My concern is low data literacy among members of CME teams responsible for writing NAs or proposals seeking funds.

It would be a more rigorous study if unique identifiers of respondents allowed a stratification of data yielding a "tidy" data analyses--"Tidy data sets are arranged such that each variable is a column and each observation (or case) is a row"--wikipedia

It appears writer's of needs assessments defer to the client or employer to designate what to include in a "first-rate" assessment. Lacking a definition of "first-rate" is also problematic. We might be comparing apples to tennis balls here.

The top 3 highest ranked choices for what should be routinely included in needs assessments as measured in survey:

medical literature review 65% (n=70)--all publications are not created equal, how is data quality measured?
learning outcomes data 45% (n=48)--need more specifics here to be able to aggregate with confidence
clinical practice guidelines 42% (n=45)--funding source of guideline? Date of guideline?

This data should have been ranked--as it tells nothing about actual behavior when aggregated. If it was ranked, we could then assign probabilities.

Writers are also asked about tables, charts, or graphics--what to include in a "typical" needs assessment. Top 3 responses: (these also should have been ranked not "select all that apply" for valuable insights).

Alignment of practice gaps with learning objectives and desired outcomes 65% (n=68)
Results of clinician survey related to practice gaps 43% (n=45)
pre test/post test data from previous activities (with to without p values) 41% (n=43)

Coming in last (behind "I don't include tables, charts, or graphics", 17% (n=18) we find "Statistics showing effect size(s) from previous learning activities", 12% (n=12).

The problem here is that charts (my collective term for visualizations) are actually little arguments. They need to be read and interpreted and if necessary, challenged--not just slapped into a document.

I think I noticed a Likert scale in the bunch but without ranking we know very little. For example, the response of 5 for one question is not comparable to a response of 5 in a different question.

I do applaud the authors of the survey (both work in CME) for efforts to measure the practice environment but I believe we left a lot of data on the table. You can read more here for thoughts on improving our approaches to CME.

When you know better, you do better (in theory)

I have worked in CME for at least 15 years or so. I consider myself a recovering CME professional. I had to step out of the bubble to notice the limits of our specialized and industry specific practices. Work in health economics, health policy, and biostatistics highlights the limited scope of "analyses" integrated in design, development, implementation, and evaluation in CME.

If anything, the role of CME is even more critical as medical education frameworks evolve to address the shifting healthcare landscape. Providers are expected to negotiate health policy, health economics, and the rigor of clinical research in decisions made at the point of care. The patient is now a stakeholder and must be informed regarding risk/benefits of treatment options and impact of care on quality of life. Community resources are also engaged in many consultations to aid in improving patient outcomes challenged by social determinants of health. That's a lot of heavy lifting for a poorly constructed multiple choice question.

As fee-for-service is replaced by value-based healthcare the traditional educational structures must evolve along with the architecture of outcomes and assessment. Most importantly, we need to define “value” based on each stakeholder.

There are trade-offs especially in the evolving age of patient-centered care. Insights gleaned from population health are informative but critical thinking at the point of care requires the ability to synthesize multiple scenarios and outcomes both clinical and economic, risk benefits and cognitive biases and heuristics. Our challenge is to embrace the integrative thinking that allows us to consider
assumptions, current reality, and critically assessed practices all together.

Too often we focus on one element or we defend assumptions as truth in stark either–or terms. We often set up false dichotomies when the most meaningful solutions lay between polar positions. I am talking to you writer of multiple choice questions to evaluate provider competencies and practice behavior.

The increasing complexity of practice necessitates the development, refinement, and application of additional and better assessment methods. These opportunities must include an emphasis on competencies beyond medical knowledge and basic clinical skills, such as systems thinking, quality improvement, interprofessional teamwork, and patient safety, while concomitantly attending to identity formation, wellness, and resilience.

In the modern age of big data we have access to control groups generated from American Community Survey Census Data, real gaps in care reported within the National Practitioner Data Bank, and limitless sources of data pointing toward patient outcomes such as MEDdra and FDA databases. Why are we still recycling outdated insights or industry funded guidelines as primary sources of information to guide care and improve outcomes at the point of care?

One way to measure treatment benefits and patient preferences for treatment is through the use of a stated preference discrete choice experiment (DCE) survey.These surveys are designed to understand the strength of preference for the different constituents or attributes of a treatment intervention or care service.

The choice experiment methodology is a powerful means of capturing patient, caregiver or physician strength of preference for the attributes of a treatment and makes it possible to understand the relative importance of attributes with respect to the degree to which people are willing to trade between them.

If you have questions I can be reached here...or on twitter @datamongerbonny

Inspiration is cheap, rigor is expensive...

10/24/2018

You might not need statistics. Hold on. Before you start doing cartwheels down the hall--this only applies if there is no uncertainty in your data.

Perhaps you have data collected from a continuing medical education activity. You are only interested in how the attendees navigated through your course and what your data demonstrates about this group and this group only.

But what if you want to be able to extrapolate these findings to the broader population? Now we have introduced uncertainty. It is impossible to reach every single member of your target audience roaming around the planet. Now we are dealing with uncertainty and the mathematics needed to talk in probabilities--statistics--"inferring proportions in a whole from those in a representative sample"

How would primary care physicians, for example, that did not attend your educational activity answer the questions you asked throughout the engagement? Better yet, what attributes of your activity are members of the educational audience accessing at the point of care? Was it your activity that nudges a participant toward evidence-based behavior? Are you certain it wasn't another program or resource, colleague conversation, alternative type of media, or other fractional or incremental activity?

I primarily focus on ideas. First, looking at wide and shallow datasets and mining for hypotheses. These are only for inspiration. The rigor is downstream of the initial queries. Collaboratively a question is identified and the team finds the best tools for the job.

Survey design is often needed to gather data.

Does your team include someone with the skillset to carefully perform mathematical approximations for continuous data, categorical data, sliding scale data? How do you determine number of questions, correct for the number of questions, construct survey to minimize response bias, and satisficing?

Who evaluates the original purpose of survey? Is a survey the best tool? What is worth measuring?

In my case, the toolbox includes R language, Python, SAS, Qualtrics, and Tableau.

The biggest problem in collaboration usually rests around a Type III or IV error.

Type I error (incorrectly rejecting the null hypothesis)
Type II error (not rejecting the null when you should)
Type III error (you correctly reject the null hypothesis, but for the wrong reason)--Statistics How To

Type IV error is a specific type of Type III error (correctly reject the null hypothesis, but make a mistake interpreting the results)

A few examples:

Aggregation bias (the wrong assumption “what is true for the group is true for the individual”--think Simpson's paradox or ecological fallacies
Running the wrong test for your data

Data clients typically invest or inherit large datasets. Perhaps they purchased an extract of claims data, CMS data, or hospital system data (intentionally vague to not reveal client identities).

Here is the caution.

Many business teams are guilty of mining their data for inspiration. If it stopped there--we would be okay. The problem exists when the attempt is made to introduce statistical "rigor" into the same data set.

Cassie Kozyrkov, Chief Decision Scientist at Google Cloud does a great job clarifying why this is problematic. Think of sitting down in the morning with your cup of tea looking forward to a nice piece of sourdough toast. The toaster pops up and holy cow, your toast appears to have the image of Elvis carefully outlined on the lightly browned surface.

If you want to investigate the miracle--or better yet--the exploits of your toaster, would you put in a fresh slice of bread or put the Elvis toast back in the toaster?

Inspiration is cheap, rigor is expensive--Cassie Kozyrkov

(data)modeling--how much crap can you take?

10/15/2018

A lot of modeling is how much crap you can take--Lauren Hutton

My first gig working in pharma was a lesson in acronyms. I had a 12 month contract at GSK and I learned an exponential amount. Everyone used verbal shorthand and I jotted the scattered letters down to look up later.

I think it softened my skin. I have no problem asking the questions. The biggest surprise is how few of us challenge the answers.

Although I have experience as a bench scientist, academic writer at a large academic medical center, as well as a lot of agency work--the added layer of pharmaceutical experience was the real hat trick.

Starting my data journey, I often heard statisticians discussing data models as if they were a thing I could pull from a shelf. They sound vetted and official don't they?

A data model is quite simply an "accumulated set of discovered relationships" within a dataset. Make no mistake the models are tested, but if sufficient "target variables" or "class labels" are not identified and considered, you can imagine the limit of insights to be gleaned from analyses.

Approached without care, data mining can reproduce existing patterns of discrimination, inherit the prejudice of prior decision makers, or simply reflect the widespread biases that persist in society.

It can even have the perverse result of exacerbating existing inequalities by suggesting that historically disadvantaged groups actually deserve less favorable treatment.--Big Data's Disparate Impact Barocas & Selbst

Working in health policy and accessing databases filtered by race, economics, and a wide variety of demographic target variables--it is critical to consider all of the "possible values of the target variable into mutually exclusive categories".

A 360 perspective is imperative while data mining. The ability to target variables specified and parsed to avoid systematically disadvantaging protected classes leads to actionable insights. It is a necessarily subjective process but can be improved with data literacy and applied to non-binary classifications.

Here is an important graphic from The Senate: Affirmative Action for White People published in the NY Times--written by David Leonhardt. Civics lesson rewind: residents of Washington, D.C., and Puerto Rico do not have voting representation in the Senate.

Right now, about four million American citizens have almost no congressional voting power, not even the diluted power of Californians or Texans. Of these four million people — these citizens denied representative democracy — more than 90 percent are black or Hispanic.

They are, of course, the residents of Washington, D.C., and Puerto Rico. Almost half of Washington’s residents are black, and nearly all of Puerto Rico’s are Hispanic.

“Big Data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.” – Dan Ariely

Walking down New Hampshire Avenue on my way back to my hotel, I stumbled upon the Democracy Tree. It is nestled between the roots of this majestic tree. Here is what it says:

This Tree is dedicated to the more than half million veterans, taxpayers, and citizens of the District of Columbia who, despite fighting in foreign wars, paying their full measure of taxes and continue to have no voting representation in the Congress of the United States of America.

"Taxation without Representation is Tyranny."
Foundry Democracy Project, Foundry United Methodist Church

The results are pretty outrageous. The Senate gives the average black American only 75 percent as much representation as the average white American. The average Asian-American has 72 percent as much representation as a white person. And the average Hispanic American? Only 55 percent as much. That’s right — the structure of the United States Senate treats a Hispanic citizen as only about half as important as a white citizen.

My point in all of this is surprisingly not political. The illustration here is to show that technology and advancements in our Big Data culture are forgetting the nuance of machine learning, AI, and algorithms--the insights are only as good as the data we feed into them.

"Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers."

The full quote below is an important example of how stewards of data (and aren't we all) have a unique responsibility to be thorough and informed as we explore and mine data. Garbage in, garbage out but worse yet--discrimination in, discrimination out...

Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with.
Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society.
Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court.--Big Data's Disparate Impact Barocas & Selbst

<<Previous

Browse the archive...

Follow @datamongerbonny

Thank you for making a donution!

Snackable content for your business needs...

I am a medical/health economics writer/ data analyst, ultra-runner, and mom.

In a world of "evidence-based" medicine I am a bigger fan of practice-based evidence.

Question the quality of the evidence. The motivation for disseminating the evidence.

Who stands to benefit the most from its uptake?

Remember the quote by Upton Sinclair...

“It is difficult to get a man to understand something, when his salary depends upon his not understanding it!”

Tweet to @datamongerbonny

The Curse of Bigness by Christopher Ketcham

When you know better, you do better (in theory)

Snackable content for your business needs...

Data and Donuts has a free weekly newsletter!

Categories