Sample Size Matters!
by Steve Boespflug
“What sample size do you recommend?” This is the most common question I hear from clients who want to do quantitative research. My usual response (which my parents became tired of when I was a kid) involves answering a question with more questions. These questions typically resemble the following:
- How large is the population (i.e., the targeted market) that will be represented by the sample?
- What segments of the population do you want to ensure are adequately represented? (e.g., region, age, income, industry, revenue, product used, etc.)
- In the final analysis, do you want to identify differences by segment? If so, what segments would you expect to compare?
The purpose of sampling is to collect, calculate and analyze statistics so that we can make inferences from the sample to the population. In the end, we need to be confident that the results adequately represent the population we care about, and that proportion differences between segments are statistically reliable. The sticking point in the sample size decision is balancing cost and reliability. In other words, choosing a sample size that’s large enough for data reliability, but not so large that clients spend more than necessary.
Many researchers determine final sample size based on standard error margins only. These are the “margin of error” or “plus-or-minus” statistics we see in the news, such as when political survey results are made public. The following matrix shows the potential margins of standard error we can expect from three different sample sizes.
Sample Size |
Margin of Standard Error |
250 |
+/- 6.2% |
400 |
+/- 4.9% |
650 |
+/- 3.8% |
To use an example, let’s say the results of a survey showed 50% of 400 randomly sampled people with smart phones generally feel safe using their smart phone for point of sale purchases. Based on standard error calculations, we should be able to say we are 95% confident that this proportion is actually between 45.1% and 54.9% of the population of smart phone users. The most important thing to consider in this statement, however, is whether the survey adequately represented “the population of smart phone users.” Did the sample properly represent the demographics of the population? Did we consider that the propensity to purchase items using a smart phone likely differs between younger and older people, or between more wealthy and less wealthy people?
When we start considering the various segments (subgroups) within the population and differences in how they respond, choosing the right sample size becomes even more important. We need to consider more than just the standard error margin on the total sample size. You see, the standard error calculation is a universally standard way of determining the potential error that will occur by not measuring the entire population. In reality, it doesn’t show us the whole picture. The calculation doesn’t specifically take into account certain biases—ultimately controllable to an extent by research practitioners—that can impact data reliability many times more. These include:
- Sampling error caused by incorrect sampling or the sample isn’t truly representative of the population.
- Question bias caused by formulating leading or incomplete questions in the questionnaire, improperly worded questions, and/or neglecting response order or question sequence bias.
- Interviewing bias caused by poor supervision or training of interviewers or failing to ensure optimal consistency across interviewing styles.
- Non-Response error caused when efforts are not made to reach and engage individuals who initially fail to respond to the survey. For example, when the sample is made up of mainly “easy-to-reach” individuals such as homemakers or the elderly who are usually at home and answer the telephone.
Closing Points
We should all be aware there is more to consider than total sample size alone and the “plus-or-minus” margins of error. Specifically:
- When designing a quantitative study, choosing a “total” sample size requires a more detailed discussion than most people realize. You must account for the important segments (subgroups) including, at the very least, the demographics (consumers) and firmographics (businesses) of your target markets.
- Good researchers control and minimize biases and error types not necessarily accounted for in the “standard margin of error” calculations. Make sure your researcher is doing the same.
Sample size does matter. We must all perform due diligence to ensure our clients understand it thoroughly and choose the right sample size for maximum reliability without spending more than is necessary. And clients, please bear with us when we answer your questions with more questions!