Beware Agenda-Based Research

A feature of the marketing and customer industry(s) is a penchant for agenda-based “research”. Time and again, it generates claims that should trigger immediate suspicion. In July 2023, Barclays Corporate Banking gave us a prime example. 

It released a report titled “What’s in Store in Retail?”, which claimed that 70% of consumers believe sustainability is a determinant in where they choose to shop. 

As a survey of 600 senior retailers and 2,000 British consumers, it was a weighty sample. But as is all-too-common, the questions seem to be designed for a target result. In this case, consumers were asked to say which of six environmental, social, and governance (ESG) factors presented to them were material to their decisions about where they choose to shop. 

In court, that’s called “leading the witness”. 

It’s a bit like asking toddlers if they want a peach, apple, or banana and then, despite never offering them alternatives such as potato chips, reporting that 70% of toddlers prefer to eat a banana for their afternoon snack, or that 100% of toddlers want fruit. 

That’s precisely what this type of supposed research does.

In the diagnostics pillar of marketing (module 3 of the Mini MBA in Customering), a key principle of market research is that we must first use qualitative data to identify the variables, and only then move to quantitative surveys to measure their materiality, i.e. – how much of the market is this trait present in? 

Agenda-based research typically skips qual, replaces it with its own biased value system (leading the witness), and goes straight to quant. In Barclays case, that was sustainability. The findings are compromised from the get-go, all in support of a pre-defined, company-desired proposition. 

Marketing professor Mark Ritson wrote about Barclays report for Marketing Week:

“The results portray UK shoppers as remarkably attuned to all the various ESG issues. Two-thirds of the population are driven to shop at locations because of all these factors. Several of them – treatment of staff, treatment of suppliers – represent a driving factor for three-quarters of British shoppers. Wow! 

‍The results do seem a little at odds with the reality of high street shopping. If the fair treatment of staff is such a driver for purchase, how come Amazon “we time your piss breaks” UK Ltd is one of the most popular retailers in the country? If supplier treatment is also top of the list, how come Tesco – “we squeeze suppliers to keep prices down” – is so huge? And if support for local communities is so incredibly strong why does US private equity-owned Morrisons do so well? Based on this data, shouldn’t most British shoppers be avoiding all these top 10 retailers and shopping at the Co-op and Oxfam instead?”

In fact, Ritson provided a contrast using research from Kantar of over 100,000 people, based on actual purchases and analysis of respondent verbatim comments instead of pre-supplied notional themes. 

This found that environmental issues were only mentioned by 4%, ethical issues by 6%, and sourcing by 1%! Barclays other sustainability factors didn’t rate at all.

Another example comes from a software company, which released a study in 2023 into ‘the state of trust among consumers. Sounds great, except for the fact that it was based on asking people to self-report on their level of, and perceptions of, trust. 

The academic, Sofianos, has done some work in this, and he summarized, “biases and interpretation limitations make self-reported trust measures unreliable”.

In fact, the literature on the unreliability of self-reporting in general is vast. There are simply so many issues – from the workings of cognition and memory to divergent personality traits to question language and onto contextual settings etc. – that there can be no endorsement of the practice. 

If you want more science on that, you’ll find it in module 1 of the Mini MBA in Customering, though it’s (sort of) summed up Margaret Mead in her now famous quote:

“What people think they do, what they say they do, and what they actually do, are entirely different things”.

That company’s “research” was, of course, not research at all, but product marketing. It served as a contrived and uber-simplistic premise that companies should “listen” to customers to ensure trust – undisguised sales code for ‘send more surveys!

It is important for marketers and customer management leaders to distinguish between product marketing and actual research, and between sales slogans and management theory. 

That starts with rejecting pseudo-studies from those with something to sell.

It should also be said that vendors and consultants that do engage in bonafide scientific research, or that simply design and deliver products or services that are based on it, stand out among the sea of those that, quite demonstrably – do not.