Statistical Significance

I was advising a client on how to collect some qualitative and quantitative research. He wanted to combine all the questions into one survey.  I cautioned him about collecting too many qualitative responses, and suggested that we may want to do two separate studies.

"Won't that take forever?" he asked? I responded that it wouldn't take that long, because we wouldn't need many responses in the qualitative part.  "But that's not statistically significant!" was his reply.  He had the understanding that no matter what type of research you're doing, statistical significance meant that you needed at least 500 respondents.  We then had a good conversation about what statistical significance means.

Statistical significance is a term used to describe the confidence level with which you can use the results of your study to project how the broader population will respond.  For example, if you are doing a quantitative study to find out how many people identify positively with your brand, you may collect 1000 responses recruited to represent proportions of the population.  If 800 of those people identify positively with your brand, you may say with a high degree of confidence that approximately 80% of the population identifies positively with your brand.  There are charts that can tell you exactly what that degree of confidence is, and that is your measure of statistical significance.

But let's say that you want to understand the associations with your brand more deeply.  You want to know how it makes people feel.  You can't ask this type of information in a quantitative survey without making some assumptions, so you decide to do some in-depth interviewing to find out how your brand makes consumers feel.  In each interview, you need to have time for a longer conversation about the consumer's values.  Even if you talk to a large group of people, if you don't go deeply enough to get an understanding of their values, your confidence level for understanding what those values are is pretty low.

In following this example through, suppose that in interviews with 10 people who love your brand, we find out that it gives them a greater sense of control than your competitors' brands.  These consumers may not have said this directly, but in 10 interviews, we were able to actively think about what we learned, and draw this conclusion with a high level of confidence. 

Would we project this conclusion onto the population at large?  No.  While we have a high level of confidence that we know the issue, we don't have a high level of confidence that this issue is true for the broader population.  For that we need a quantitative study with a large sample size.

What does the term "statistical significance" mean at your company?  Remember that it is a measure of your level of confidence that you have the right answer.  This will vary based on what you are trying to learn, the size of your total population, and how deep you need to dig for answers.  If you are doing very in-depth qualitative research, remember this:  A large sample size may be less statistically significant, because you won't be confident that you could find the right answer in the first place.