In the olden days – and by “olden” I mean the last century – it was expensive to conduct market research. Even if you only asked a single question you were generally looking at spending at least $1,500. Today, you can ask a single question, and get a heap of profiling data, for around $100. And, you get your results much faster.
To get a feeling for how this works in practice, I road-tested a couple of the new super-cheap services: Toluna QuickSurveys and Google Consumer Surveys. I tested Google because it is the new kid on the black, doing things completely differently to everybody else. There was no particular science behind my choice of Toluna and I suspect I would have had broadly similar experiences if using any one of the other “traditional” DIY market research services.
This post compares my experiences using the two tools. I road-tested them with a super-simple survey which asked just one question:
Using 10 words or less, how do you feel about Microsoft?
Price: Google wins
Google: $100 for 1,000 responses. Toluna: $105 for 300 responses.
Timeliness: Toluna wins
Google: 2 days. Toluna: 1 hour.
Ease of use: Toluna wins (just)
This was hard one to judge. Both of the programs are marvels. If you know what you want to ask and who you want to interview you can do all your set up in a few minutes. Both services are exceptionally easy to use.
Unfortunately, my questionnaire for Google went out with a small error in it. The phrase “Not at all likely” appeared in grey text just beneath my question. While this was undoubtedly my mistake, it was a mistake that was, to my mind, caused by some some errors in Google’s user interface design. In particular, whenever I discovered that Google could not do what I wanted (more about this below), I had to go back to the beginning. So, to save some time I did a whole lot of cutting and pasting, and this led to my error. What added to this was that Google never gave me a preview of my questionnaire, so I was never forced to confront my error. And, somewhat irritatingly, I realized my error before Google had started conducting the interviews but Google did not let me fix the problem.
With Toluna my challenge was a bit different. The interface is less intuitive than Google’s. While it is very simple to use, I regularly found myself thinking “What should I click now?” and “What do these strange icons mean?”. Having said that, it still only took a few minutes to get my questionnaire programmed, so it was not super-confusing. As my Toluna survey went out without errors, I score Toluna as the winner in terms of ease of use. However, this
Asking what you want to ask: Toluna wins
Google forces you to ask very short and simple questions. In principle, this is a good feature. Short and simple questions typically result in better quality data. However, Google is too restrictive and it can lead you to do silly things. For example, consider the ubiquitous the Net Promoter Score question, used by businesses around the world to benchmark performance. I entered this into Google just by typing it as I wanted it, on the boxes on the left of the screen shown below. The preview is on the right. Note that the question Google previews is not the one I am asking. Google has automatically decided to only show five of the options and randomly ordered them. The resulting question makes absolutely no sense. While there is an option saying Randomize answer order, it was greyed out and I could not change it. The closest I could get to the question that I wanted to ask was to give people a rating out of five or seven stars. The error that I made in my survey was a direct consequence of this problem. Because there was no way of changing the type of question from Single answer to other question types I had to do a lot of cutting and pasting, and I made the silly mistake. As I said above, it was my mistake. However, a key principle of designing software is that it should be forgiving, allowing you to modify your choices rather than forcing you to go back to the beginning, so ultimately my error was a consequence of Google’s user interface design.
With Toluna QuickSurveys you can easily ask all the standard market research type questions, which places it a long way ahead of Google.
Analytics: Google wins
Toluna lists the individual responses and provides a very basic word cloud:
Google’s analytics are great. They are a bit complex to get your head around initially, but once you have invested the time they are easy to use. The word cloud is even better than it looks (it is almost as good as DataCracker‘s), because Google has automatically grouped together synonyms (e.g., good, great and nice). And, if you want you can get Google to instead look at Sentiment. Under the word cloud Google is showing the most common words and comparing them by gender. You can also choose to compare by inferred age, region, population density and income.
Data file: Toluna wins
Both Google and Toluna allow you to download a CSV data file. The Google data contains some demographics: Gender, Age, Geography, Urban Density, and Income. The gender and age data is “inferred” and incomplete (e.g., about a third of my sample had missing age data). The income data looks highly suspect and is presumably inferred as well.
Toluna data file comes with more than twice as much demographic data: Gender, Age, Region, Ethnicity, Race, Education Level, Individual Annual Income, Work Position, Primary Grocery Shopper, Children in HouseHold, People in HouseHold, and Province. And, to my eye anyway, it looked to be better quality.
Sample representativeness: Nobody wins
As is, has, and will always be the case, the great problem with such cheap online data is the extent to which it truly reflects the market we are trying to understand. With Toluna it was easy to see that the data was not completely representative, with 41% of my respondents being female. With Google, there was no way to check at all.
Quality: Toluna wins
When I dug further into the data it became clear that the Google Data was greatly inferior to Toluna’s. The table below shows the first 10 responses from Google and Toluna. Note the brevity of the data from the Google respondents. On average, their responses were about one-third the length of the Toluna responses. The Google respondents’ data reflects their reality: they are basically forced to answer the questions in order to get access to websites they like. By contrast, the Toluna QuickSurvey respondents have volunteered to participate in the surveys, and while this may bias their data, it clearly has a huge impact in terms of the quality of the data they have provided for my question.
Toluna Quick Survey
|Stop pumping us for information!!!||They make really innovative easy to use products.|
|it’s a big corporation||good company|
|great product, good company. Anything not Apple has my vote||VERY GOOD COMPANY|
|no comment||I like them a lot|
|indifferent||Technology leader with products widely used by consumers and companies.|
|dont care||it has many tools to make computer use easier|
|it is very helpful when i am searching for information||it is a decent os|
|great||I think Bill Gates is a genious|
|no||Microsoft is a very well known brand that is well liked.|
|Powerful||missed the boat on mobile|
The verdict: if your needs are very simple use Google; otherwise, use Toluna
Provided that you can ask a very simple question and it is OK that people only pay it superficial attention, then Google’s Consumer Surveys is the clear winner, as it is cheaper and the analytics are much better. However, it is pretty rare to have such simple needs. For anything else Toluna QuickSurveys will end up being much better.
In addition to the factors mentioned above, a particularly important point of difference is how the services treat surveys with multiple questions. Google basically treats them as separate studies. So, a simple question like “What proportion of the people that like my product are men?” cannot be accurately answered with Google. However, Google will give you an indicative answer to such questions, as it infers people’s genders and ages from their online behavior and includes these educated guesses in their analyses, so you can at least work out whether the people that like your product have more masculine online behaviors than people that do not like your product. But, if you are wanting to see relationships between other questions, such as looking to see if there is a relationship between a question that asks what people want and another question which asks people how they behave, you are all out of luck with Google and will be much better placed to use Toluna QuickSurveys or one of its many competitors.