I think it is important to visit statistical concepts from time to time especially when the same misconceptions are replicated from client to client. Following the inaugural health economics and outcomes research (HEOR) writing skills for medical writers workshop last week in Philadelphia my colleague presented Terminology and Jargon Demystified for Medical Writers. I thought I had a herculean task to answer the What is Value and Why Does it Matter question in the limited conference presentation time slot but in this case--I got off easy. She had to address unique methodology and statistics specific to HEOR.
Quite often when leading workshops and being precious with our planning and cinematic presentation style we all tend to forget one thing. The fundamentals are where we lose most of our colleagues.
In medical education data I notice the preference for the t-test approach. Unfortunately appropriate application of the t-test assumes that the difference between a response of 1 and a response of 2 is exactly the same as a difference between a response of 4 and a response of 5, for example. This is not usually a reasonable assumption but it is one made quite often.
The most appropriate test for comparing pre- and post- data on a Likert scale is typically McNemar's test. To further complicate the analytics, there are assumptions often applied but perhaps not actually relevant. For example, we all assume that the difference in the scores is solely due to the exposure to an intervention like an educational activity.
Understanding what we mean by normally distributed data is paramount to understanding the selection of appropriate tests and measures. I suggest asking a lot of questions because most of us aren't born with an inherent understanding of categorical and ordinal variables.
Khan Academy does a nice job of providing a good base or try one of the Coursera classes.
Do you have to p? Are p-values really necessary?
You often hear people quoting p values as a sign that error is unlikely. “There’s only a 1 in 10,000 chance this result arose as a statistical fluke,” they say, because they got p=0.0001. No! This ignores the base rate, and is called the base rate fallacy. Remember how p values are defined:
The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed.A p value is calculated under the assumption that the medication does not work and tells us the probability of obtaining the data we did, or data more extreme than it. It does nottell us the chance the medication is effective.
When someone uses their p values to say they’re probably right, remember this. Their study’s probability of error is almost certainly much higher. In fields where most tested hypotheses are false, like early drug trials (most early drugs don’t make it through trials), it’s likely that most “statistically significant” results with p<0.05 are actually flukes.
--Statistics Done Wrong
This is a topic we will return to over and over again--misconceptions in data analytics and information to help un-muddy the waters. Shoot me an email or a comment if you have specific questions.