Do you have to p? Are p-values really necessary?
Because the base rate of effective cancer drugs is so low – only 10% of our hundred trial drugs actually work – most of the tested drugs do not work, and we have many opportunities for false positives. If I had the bad fortune of possessing a truckload of completely ineffective medicines, giving a base rate of 0%, there is a 0% chance that any statistically significant result is true. Nevertheless, I will get a p<0.05 result for 5% of the drugs in the truck.
You often hear people quoting p values as a sign that error is unlikely. “There’s only a 1 in 10,000 chance this result arose as a statistical fluke,” they say, because they got p=0.0001. No! This ignores the base rate, and is called the base rate fallacy. Remember how p values are defined:
The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed.A p value is calculated under the assumption that the medication does not work and tells us the probability of obtaining the data we did, or data more extreme than it. It does nottell us the chance the medication is effective.
When someone uses their p values to say they’re probably right, remember this. Their study’s probability of error is almost certainly much higher. In fields where most tested hypotheses are false, like early drug trials (most early drugs don’t make it through trials), it’s likely that most “statistically significant” results with p<0.05 are actually flukes.
--Statistics Done Wrong
Those poor p-values. They sort of got swept up in the statistical fervor as an easy concept to validate research findings. They didn't mean to mislead but many of us look at data whether in our own analytics or just scouring peer-reviewed research and are often astounded with the misconception about their significance (pun intended). Remember anyway--statistical significance doesn't often translate to clinical significance.
This is a topic we will return to over and over again--misconceptions in data analytics and information to help un-muddy the waters. Shoot me an email or a comment if you have specific questions.
Sign up for our newsletter!
Browse the archive...
Thank you for making a donution!
In a world of "evidence-based" medicine I am a bigger fan of practice-based evidence.
Remember the quote by Upton Sinclair...
“It is difficult to get a man to understand something, when his salary depends upon his not understanding it!”