Market Research. Is There Too Much of It?
Change. The one variable that is constant. And usually, change means better. No one changes to something that was worse than what they had before. For those who hanker for the “good old days”, think about what it would have been like to go to the dentist 100 years ago?
Days of Innocence
Way back in 1981, when I first joined a market research firm fresh out of University, I was engaged initially as a mathematician to ensure proper random sampling.
Research firms had their own field force back then and most quantitative surveys were done door-to-door, face-to-face. We worked from Collection Districts from the ABS. A major issue was cost – an interviewer was given a strict set of dwellings and if nobody, or the appropriate person, was not home, they were to try 3 times before replacement a dwelling was allowed.
Decent sample size, quantitative studies were too expensive for all but the large companies. Anecdotal findings, coupled with experience, were often all small businesses had.
There was also the obscene amount of paper used – imagine a 6-page questionnaire for 1,000 respondents.
The number of successful interviews achieved in a day was “bugger all” and a sample of 500 or 1,000 took ages just to complete the field work. And of course, the cost was often seen as prohibitive for the perceived results. (That the results many times reinforced what the marketing team believed they already knew, was deemed a waste of money).
In the early 80’s there were not any PCs – coded questionnaires (coding is a valuable art that has been lost) were taken to a company who produced a computer print out. Often, after working through the first run, additional reports were produced – “Can we see how the breakdown how of respondents in question 4 answered question 6?”. On top of this any statistical analysis was done separately, again after all report runs had been completed.
Often, the alternative was “Let’s just run a few groups”. Great, an-depth discussions with 30 odd people was not really an accurate indication of anything, bar the lack of understanding of how research was best used.
Many clients would balk at spending $30 to $50,000 to optimise a $5 million campaign.
By 1983 I had graduated to running groups and have fond memories of running focus groups in people’s kitchens. The respondents were genuinely thrilled that large consumer companies sought out their opinions and engaged in discussions accordingly.
It was a novelty and as such they gave honest answers and when the topic was exhausted, that was it, the group was over (easy to spot). Yes, they were paid, but many times I had respondents state they would have participated even if not paid.
Many golden nuggets of consumer thinking (and statements, or in today’s terminology, content) were gleaned from them.
Cost and Technology. Agents of Change
Telephone surveys took over from door-to-door sampling – nowhere near the cost of sending interviewers out into the field.
For market researchers, their most valuable natural resource is respondents. Telephone interviews soon became problematic thanks to telemarketers.
“Hi, I am doing a survey on house cladding”. Several question later they knew your type of dwelling and if you were a candidate for cladding. And in no-time there was a knock at the door, or sometimes a phone call, from a person trying to sell you cladding.
And as telemarketing grew, many people had experienced having their dinner disturbed by someone trying to sell them something, camouflaged as market research.
“Piss off” became a frequent response. Many households grew tired of receiving telephone calls, be they from a genuine research organisation or a telemarketer.
By this time, PCs were in all research companies and the questionnaires were on screen. As the interviewer “filled in” the questionnaire on screen, the first run of data was being done simultaneously. Not anything to marvel at today, but when this first became available, it was revolutionary and research companies tried to push all quantitative surveys into telephone surveys.
But all the while, researchers most valuable natural resource, willing and uncorrupted respondents, was gradually being eaten away.
Who has not been Surveyed?
Fast forward to 2019 and I doubt there is a person over the age of 18 who has not been surveyed in some form. As more and more businesses move online, they see it as an opportunity to tag a few research questions at the end.
The quality of the research results is totally dependent upon the quality (accuracy) of the sample – is it representative of the group you want to survey? “Normally distributed, proportionate to size”, was a phrase used in every quantitative study back when door-to-door was the norm. The sample would be “reweighted” to the exact proportions of the various demographic groups in the population being studied.
Research is the same as computers i.e. rubbish in = rubbish out.
I have no doubt that reputable research firms adhere to strict sampling disciplines, but this does not solve the problem of “too much research”.
I had lunch recently with 2 friends who have nearly 80 years of market research experience between. They felt that too much research kills our natural resource – the unbiased and honest respondent. Our chat formed the genesis of this article.
Many large research companies today run Panels – tens of thousands of people who, for a reward (often cash), are willing to be respondents. By having all the relevant demographic information of each panel member and by filtering via certain behavioural questions, they can supply a ready-made sample.
But just how random is this group of people? They can distort focus group discussions – a couple may attend up to 2 groups each a month for $80 (or more) per group. And they are also paid to participate in quantitative studies. This can give them up to even $320 a month extra cash in hand, nothing to be sneezed at.
They are paid to give an opinion and so they do, but whether this is what they really believe, or what they think they should say, no one can truly tell.
“Do you buy jam?” – “Yes!”
“What brand did you last buy?”. The brand manager for Cottee’s jams is deeply involved in the brand, but the average shopper only gives it a cursory consideration when shopping. “Last brand? Um, Cottee’s”. It may not be, but as they are paid for an opinion, they better have one. It could have just as easily been another brand that was on special. But Cottee’s is top of mind and gets the vote.
7 Sounds like a Good Number
And then there is the classic “How likely would you be to recommend…….?” The Nett Promoter Score. I read not long ago the new MD of a large private health insurance company was going to link all senior executive’s bonuses to Nett Promoter Scores. Poor bastards!
Mark Ritson wrote a column recently about CEOs’ messing up marketing and advertising – what a colleague refers to as “Seagull Management” – they fly in, shit over everything and fly off.
This was probably a good example. The MD had probably heard of The Nett Promoter score and thought “Bloody brilliant. I will introduce that”.
Where once a survey was something rare and genuinely regarded as being worthwhile, people are asked to rate something nearly every day. And hence we have “7 sounds like a good number”. Nett Promoter Scores may have been worthwhile years ago before the natural resource of “virgin” or at least “rarely used respondents” had been used up.
Now, people rattle off numbers between 5 and 7, unless something extraordinarily good or bad had happened to them.
A/B testing is another hoary chestnut. Not the place to detail an in-depth dissertation on the problems with this technique. It has become so commonly used that I would prefer to take the opinion of an experienced and intuitive creative person to finalise a headline or ad concept, over the answer given by some spreadsheet A/B testing technique.
Survey Monkey has allowed smaller businesses to conduct their own research. Sometimes this does throw up useful information, but often it is conducted by people with no experience in sampling and interviewing. The analogy is of the person who acts as their own legal counsel has a fool for a client.
We know that thanks to Google, Social Media and almost no end of information sources, consumes are far better informed than ever.
I have seen focus groups in which the respondents were acting more as creative directors – “It would have been more impactful if you had used morphing”.
Testing communications against their objectives is becoming more and more testing.
Data seems to have usurped research as a source of insights. In many companies, the market research and data insights are separate departments. Where once we referred to data as Primary Research – all available information (data) was analysed prior to any market research being developed – what do we know, or think we know, what gaps are there is our knowledge etc. Market research and data were all part of marketing intelligence. There was no demarcation between the two, they worked together seamlessly.
Is there an Answer?
There have been many fantastic new research techniques and programmes developed over the past 40 years. That is a given. But it does not address the single biggest problem facing any form of market research. As with data, quality should always take precedence over quantity.
No matter how good the technique, if the sample is a dud, then the results will turn out the same. Nielsen introduced, a number of years ago, a qualitative research technique they called “Delta Qual”. Derived from forensic psychology, its aim was to determine what people actually did when grocery shopping, not what they thought they did.
Not a simple or cheap technique, it required 2 moderators. Though this approach is more likely than many techniques to find the truth, it is still heavily dependent upon the quality of the respondent.
“Killing the goose that laid the golden egg”, was how one researcher described the over use of research.
This is a topic in which it is far easier to identify the problem than it is to provide an answer. Cliched as it is, the answer is “each case is different”.
Prior to any research study, be aware of the fact that many of the respondents are jagged and give an answer off the top of their head. If you have received feedback from say 5,000 people and 4,000 of them are just going through the motions, then the results will be rubbish.
How to identify and access the respondents who are genuinely interested in providing answers? Of course, this then raises the problem of statistical validity. Make sure you know the demographic and relevant behavioural breakdown of your target group and then reweigh the results accordingly.
With qualitative research, speak to the people who are either very happy or very pissed off.
Have you ever been to a restaurant where the waiter or waitress continuously interrupts to ask, “How is everything”? Maybe we need to learn from this. Stop asking questions after every customer contact and learn to identify those who really do have something to say.
First published in B&T – July 2019.
