Perhaps you have data collected from a continuing medical education activity. You are only interested in how the attendees navigated through your course and what your data demonstrates about this group and this group only.
But what if you want to be able to extrapolate these findings to the broader population? Now we have introduced uncertainty. It is impossible to reach every single member of your target audience roaming around the planet. Now we are dealing with uncertainty and the mathematics needed to talk in probabilities--statistics--"inferring proportions in a whole from those in a representative sample"
How would primary care physicians, for example, that did not attend your educational activity answer the questions you asked throughout the engagement? Better yet, what attributes of your activity are members of the educational audience accessing at the point of care? Was it your activity that nudges a participant toward evidence-based behavior? Are you certain it wasn't another program or resource, colleague conversation, alternative type of media, or other fractional or incremental activity?
Survey design is often needed to gather data.
Does your team include someone with the skillset to carefully perform mathematical approximations for continuous data, categorical data, sliding scale data? How do you determine number of questions, correct for the number of questions, construct survey to minimize response bias, and satisficing?
Who evaluates the original purpose of survey? Is a survey the best tool? What is worth measuring?
In my case, the toolbox includes R language, Python, SAS, Qualtrics, and Tableau.
Type I error (incorrectly rejecting the null hypothesis)
Type II error (not rejecting the null when you should)
Type III error (you correctly reject the null hypothesis, but for the wrong reason)--Statistics How To
Type IV error is a specific type of Type III error (correctly reject the null hypothesis, but make a mistake interpreting the results)
A few examples:
- Aggregation bias (the wrong assumption “what is true for the group is true for the individual”--think Simpson's paradox or ecological fallacies
- Running the wrong test for your data
Here is the caution.
Many business teams are guilty of mining their data for inspiration. If it stopped there--we would be okay. The problem exists when the attempt is made to introduce statistical "rigor" into the same data set.
Cassie Kozyrkov, Chief Decision Scientist at Google Cloud does a great job clarifying why this is problematic. Think of sitting down in the morning with your cup of tea looking forward to a nice piece of sourdough toast. The toaster pops up and holy cow, your toast appears to have the image of Elvis carefully outlined on the lightly browned surface.
If you want to investigate the miracle--or better yet--the exploits of your toaster, would you put in a fresh slice of bread or put the Elvis toast back in the toaster?
Inspiration is cheap, rigor is expensive--Cassie Kozyrkov