1. What is a Qualtrics Online Sample?
Qualtrics Online Sample is a way to get a more-or-less representative sample of almost any particularly demographic group from most countries in the world.
2. Is it really a representative sample? Sort of.
Technically the Qualtrics Online Sample is not a true random sample, but rather a quota sample.
A true random sample (i.e. mathematically random, not just haphazard/convenience) requires respondents to be drawn from a list (called ‘sampling frame’) of all (or practically all) members of the population. True random samples are expensive and rarely feasable.
Instead qualtrics useing a quota sampling method, where they attempt to make the sample they give you share the same demographic characteristics as the population you are trying to study. So, for example, your survey of ‘Australians over the age of 18 years of age’ will have the same proportion of people of the same age, income, gender, education, and approximate geographic distribution, as the Australian population. The advantage of this is that the method is cheap - Quatrics recruits people who want to be subjects for market research, and then selects those that fit the profile you need. The disadvantage is that you have an unaccounted for selection bias in the sample - the type of people who choose to do qualtrics surveys are the only people you can survey, and it is hard to know what biases might be contained in such as sample. For example, Qualtrics Online Samples will definitely include people with access to the internet, and also people who are more likely to answer yes to the question “Would you like to participate in market research?”
3. How much does it cost?
As at the time of writing this (September 2019) the cost of representative samples of the Australia population are:
| Length of survey | Cost per response |
|---|---|
| 10 min or less | AUD$6.50 |
| 15 min | $7.50 |
| 20 min | $8.50 |
| 25 min | $9.50 |
| 30 min | $10.50 |
| 35 min (max) | $11.50 |
Note on currency conversion for those outside Australia:
- 1 australian dollar (AUD) = 93c Singapore (SGD) (30/8/3019)
- 1 australian dollar (AUD) = 67c US dollars (USD) (30/8/3019)
The pricing formula is AUD$6.50 for first 10 mins, and then AUD$1/each extra 5 minutes.
35 minutes is the maximum length of time recommended by Qualtrics, as their analysis of surveys shows that data quality drops dramatically after this point (a result of what is technically called ‘respondent burden’ - i.e. exhaustion and bordom).
4. Other advantages
Qualtrics Online Samples have a number of advantages for researchers:
- Demographic data for free: The data you receive from Qualtrics will include a range of demographic data about each respondant: age, income, job, location, gender, education. This has two big advantages. First, you have this data, which is often useful for including in publications. And second, you don’t need to actually ask questions (and therefore spend respondent time, and therefore money) on asking these questions to respondants. Qualtrics recommends that if the variables are very important to your study, then you ask for them during the survey. However, for those on tight budgets, or who want to use the time they have for other questions, this can be a good way to get more data from a single survey.
- Any country: Almost any country in the world can be surveyed. Qualtrics has respondants in almost any country, except places like North Korea. You can get a quote for replicating your survey in any of these countries. The cost of these will likely be different to those above. Qualtrics also offers translation services for translation of your survey into other languages (around 90c/word, but discounts available for large orders)
- Specific Populations: You can also ask qualtrics for specific populations, such as young people in Australia, or NSW. The cost of these is likely different to that listed above.
- Only pay for quality responses: You only pay for quality data. So if you get nonsense responses, then these will be filtered out by Qualtrics and if you find any others, they will replace them for free. When you pay for a respondent you only pay for complete, quality answers.
- Ethics: Qualtrics apparently provides a set of advice about how to answer common questions raised by ethics committees/institutional review boards. I will post this when I have access. I am trying to get details on how much is paid to respondants, and will post that when I get confirmation of this.
5. Alternatives to Qualtrics Online Surveys
The most popular alternative to Qualtrics Online Samples is probably Mechanical Turk, a service offered by Amazon.
However, there are a large number of other providers which you can (probably) easily find with a google search for “online survey respondents” or “alternative to MTurk for surveys”