Qualifying Market Research’s Quantitative Bias

I recently came across a new term which makes complete sense. ‘Quantification bias’ is the mistaken belief that those things that can be measured are more important than the immeasurable. It is becoming an epidemic in the world of marketing and advertising with the growing fetishisation of big data and artificial intelligence.

The idea that we can measure everything that matters is a dangerous fallacy. Market researchers have always been prone to the mistaken view that being able to ask a question means that the answers will provide insights into human behaviour, and as the availability of other data grows, quantification bias grows too. If quantification bias is true of primary data, it is even more true of passive secondary data.

Neil de Grasse Tyson said that, “In science, when human behavior enters the equation, things go non-linear. That’s why physics is easy and sociology is hard.” Market research is uniquely placed to answers the hard questions that only sociology, psychology, and other social sciences can help to answer.

That doesn’t mean that social science is or should ever be subjective, and the use of frameworks helps researchers interpret subjective data with objective thinking. Christian Madsbjerg’s recent book Sensemaking provides a great framework for understanding human behaviour through five key themes.

First, and most importantly, he emphasises the importance of culture over individuals. He points to the importance of sociology in a world where nearly all behaviour has social context and is not just the outcome of individual traits.

Secondly, he talks about ‘thick’ data as providing deeper insights than ‘thin’ quantitative data (including big data). All human behaviour has to be related to the greater cultural context. As he points out, a wink is a twitch of an eye lasting a millisecond, but it can mean so much. “I’m not serious” or “Let’s leave together” or “You’re an idiot” or many other meanings can be derived from the same small gesture, but without understanding of context it’s impossible to know.

“Algorithms don’t make things fair, they repeat past practices – they automate the status quo.”
Cathy O’Neil in Weapons of Math Destruction

Thirdly, he talks about the value of the savannah over the zoo, by which he means understanding behaviour in the real world rather than in an artificial setting (think of ethnographic research vs the focus group discussion). Experience and perception are relative and situated. They can’t be measured, nor can they be removed from their context.

Fourthly, he discusses the importance of creativity in research. We often talk about induction and deduction as important skills, and while Sherlock Holmes often mentioned deduction, a lot of what he practiced was abduction. Abduction is a combination of deduction, induction, and creative thinking to fill the gaps and make connections between different pieces of data. While this might be disparagingly called ‘educated guessing’ by some, I believe this makes the difference between summarising data and synthesising data (inference, not reference).

Abductive reasoning is messy and not always logical, but it is far more insightful. It requires researchers to remain in a state of doubt and hold conflicting ideas in their heads. The great value of doubt is that it opens up our minds to new understanding and creative thinking. If we know that a window is broken in a house, the jewelry is missing, and some items have been overturned, we can infer that a burglary has happened (and most of the time we will be correct).

Madsbjerg’s final theme is that the world is not as complicated as it sometimes seems, and that there remains great value in common sense. While we are often fixated on staring at the GPS on our phone, we should look up more often at the stars above our head to see the real picture (and find the right direction).

Making sense of the world needs common sense approaches. Staring at the GPS of big data can only get us so far (as I’ve often experienced with taxi drivers). Focusing on technology means that we miss the big picture and the important things to be seen by looking around. We can only know what is important if we step away from the data and see how it fits into the context and culture of the behaviour.

Albert Einstein had it right when he said, “Not everything that counts can be counted, and not everything that can be counted counts.” •

By Neil Gains, TapestryWorks