| This is one of my favorite expressions although I recently realized that the origin is far from resolved. Clearly from a culinary perspective those of us who spend far more than the minimum amount of time in the kitchen can appreciate what just the right onion can contribute to a dish. The phrase itself refers to being knowledgable in a specific area. I even included a song by the Shins for you to enjoy while you read along. On the topic of knowing your onions I also stumbled upon a useful graphic design book of the same title. My personal library is filled with the odd design and graphic design book and you would be surprised how often they come in handy when developing a brochure or a communication to be printed or even more importantly displayed in a digital format. Everything you create needs to be informed by a creative and aesthetic lens. Think about how usability has informed much of what we know about design or even interactivity in survey design, learning platforms, or to encourage engagement with your website or blog posts. Social media has driven us all to become more aware of the look and feel of the content we develop and I for one welcome the evolution. |  |
Talk data to me... |  |
In my professional work I often review and analyze continuing medical education (CME) outcomes. There is one reoccurring dilemma that I want to try and help you avoid. Now let me qualify this by saying I know how limited CME budgets can be and how many of the steps associated with survey design or question structure may seem straightforward enough that you attempt to fly solo. That's fine too but I think I can help keep your journey safe and more importantly statistically rigorous.
I think it will be easier to appreciate the guidance if you try to appreciate the difference between a problem and an issue. Once you define a specific problem it becomes easier to create a checklist of concise issues that you have the power to resolve immediately. For example, a limited budget for statistical consulting may undermine your wish list of developing strong outcomes data (problem) but limited understanding of a data framework can be resolved with a helping hand or a reference resource (issue).
Many question designers rely on evaluating percent correct on a knowledge type question. Perhaps you even integrate a case study or an informative question stem followed by a list of potential decisions or answers. Outcomes data that I recently reviewed had these question types coded as either correct or incorrect. As I read on in the reports I see that the t-test or ANOVA statistical procedures were proudly cited but there is one problem. Dichotomous data is not normally distributed and these statistical procedures require the assumption that the data are normally distributed. Inferences from your data are now invalid and do not report the findings that you had hoped for. I have encountered numerous occasions where the go-to statistical procedure was the t-test (or ANOVA) without consideration of whether it is actually the correct test to use.
When I design survey instruments I prefer a Likert approach but again this introduces additional statistical complexity. Why Likert? In medicine there is rarely a wrong or right question that can be applied to all patients. Because of the nuances of the healing arts I try to avoid labeling and therefore analyzing a response as either correct or incorrect. The most useful advice I can recommend when you are designing these elements with limited resources--clarify what you intend to do with your data. If you are interested in showing that scores differ when considering different groups of participants (primary care physician vs. specialists), you may treat your scores as numeric values, provided they fulfill usual assumptions about variance (or shape) and sample size. If instead you are interested in highlighting how response patterns vary across subgroups, then you should consider item scores as discrete choice among a set of answer options and look for log-linear modeling, ordinal logistic regression, item-response models or any other statistical model that allows to cope with polytomous items.
Stay tuned for additional insights and discussions...pick your onions carefully!
I think it will be easier to appreciate the guidance if you try to appreciate the difference between a problem and an issue. Once you define a specific problem it becomes easier to create a checklist of concise issues that you have the power to resolve immediately. For example, a limited budget for statistical consulting may undermine your wish list of developing strong outcomes data (problem) but limited understanding of a data framework can be resolved with a helping hand or a reference resource (issue).
Many question designers rely on evaluating percent correct on a knowledge type question. Perhaps you even integrate a case study or an informative question stem followed by a list of potential decisions or answers. Outcomes data that I recently reviewed had these question types coded as either correct or incorrect. As I read on in the reports I see that the t-test or ANOVA statistical procedures were proudly cited but there is one problem. Dichotomous data is not normally distributed and these statistical procedures require the assumption that the data are normally distributed. Inferences from your data are now invalid and do not report the findings that you had hoped for. I have encountered numerous occasions where the go-to statistical procedure was the t-test (or ANOVA) without consideration of whether it is actually the correct test to use.
When I design survey instruments I prefer a Likert approach but again this introduces additional statistical complexity. Why Likert? In medicine there is rarely a wrong or right question that can be applied to all patients. Because of the nuances of the healing arts I try to avoid labeling and therefore analyzing a response as either correct or incorrect. The most useful advice I can recommend when you are designing these elements with limited resources--clarify what you intend to do with your data. If you are interested in showing that scores differ when considering different groups of participants (primary care physician vs. specialists), you may treat your scores as numeric values, provided they fulfill usual assumptions about variance (or shape) and sample size. If instead you are interested in highlighting how response patterns vary across subgroups, then you should consider item scores as discrete choice among a set of answer options and look for log-linear modeling, ordinal logistic regression, item-response models or any other statistical model that allows to cope with polytomous items.
Stay tuned for additional insights and discussions...pick your onions carefully!
