The Coming Revolution in Customer Insights

With substantial investments in market research, most successful companies know 90% plus of what there is to know about their current customers—habits and practices, price sensitivity, switching behaviors, brand engagement, you name it—yet much of that knowledge is largely irrelevant when it comes to driving big innovations.

Why is that? Where did the money go?

The root of the problem with most market research is that it is directed at a current reality: the existing frame of reference, the present set of products and services, today’s users and non-users…the here and now. It goes about as deep as you can go in a known and observable domain. And there’s no question that it’s very helpful in creating incremental improvements to products and services.

When companies apply their customer knowledge base to innovation, however, that’s where they tend to go astray: they extrapolate from the current reality, assuming that behavior change is hard, so customers will want incremental change and will think and act in a rectilinear fashion.

Yet, if you think about it, all the recent successes in innovation, from Apple to Google to GPS to TiVo to hybrid cars and beyond — all represent disruptions and sudden, dramatic changes in customer behavior that were anything but straight-line projections of existing segments, products and markets.

No customer asked for these innovations when called on in a focus group, while participating in an ethnography, while filling out an online survey or chatting at a bar with an account planner. That’s exactly why Steve Jobs said, famously, “customers don’t know what they want until we’ve shown it to them.” Hubris? Sure. But Jobs knew that you can’t expect innovations to be served up piping hot in traditional market research.

Innovation is all about exploring the periphery, discovering everything that happens adjacent to the current customer experience, before and after your product is in use, around and outside the category of today. It’s about looking not just for current pain points, or places to improve, but looking for structural anomalies between customer needs and the optimal way to fulfill those needs.

It’s not about small ball. It’s about thinking outside the chalk, beyond today’s game.

So, how exactly do you conduct your customer discovery to discover the next new thing?

The short answer would be: don’t look in the same places, don’t start with your existing market, products and brands and don’t believe what people tell you.

Today, huge advances in our understanding of cognitive behavior and decision science are changing the way that we think about market research and customer insight, turning a decades-old model that depends on customers’ rational self-reflection into a much more adaptive learning environment that recognizes the inherent messiness of human thought and behavior.

We now know that the brain craves “cognitive ease,” a world of simple and repeatable patterns, easy rules of thumb that help us to make up our minds quickly and effortlessly, heeding familiar signals and avoiding the noise of different, unfamiliar stimuli.

In short, we don’t like to think too hard, be too analytical or spend much time pondering areas where doubt and uncertainty exist … that’s what science calls “ego depletion,” and people studiously avoid it.

Ask a customer a question, any question, and they will default to the easiest, most simplistic answer … keep pushing on the topic, laddering all the way! — and they will quickly contrive a rational world of logical narrative. Ask them about their emotional connection and they will give you a torrent of rich language that lays bare their inner being. Show them a new product concept and they will speak authoritatively about how it might fit into their lives.

Well, if that’s how it’s supposed to work, than why is the new product failure rate across the seven largest economic sectors in the U.S. north of 90%?

And why is it so hard for companies to state declaratively what the Top 10 unmet needs of its customers are?

The root cause of this broad failure of market research to return actionable insights that create innovation value is threefold.

Most research is:

  • Designed to be projectable, so much so that even qualitative inquiry or smaller quant surveys are based on finding representative sample sizes from which the researcher can extrapolate to large populations;
  • Designed to be confirmatory, seeking to validate existing executive hunches, rank order raw concepts for appeal and purchase intent or, most often, to rationalize the expense of a marketing activity or the extendibility of a brand;
  • Too gullible, believing what customers say … even though this is not what science tells us we should believe.

Here are five simple pathways to break out of the current market research model and to jump start the kind of customer discovery that leads to true innovation:

Rule #1: “small n, Big P”

Focus on a deep dive with 4 to 6 customers over an extended period of time. Resist the temptation to canvass large populations or to try and find representative samples (representative of what is a big question, and really hard to answer when it comes to disruptive innovation). Worry a little less about geographic distinctions or demographic cohorts. The right people will give you disproportionately higher and more actionable input.

It has been demonstrated conclusively that a small set of respondents will yield most of the available insights in any given area of discovery, and that the marginal return on adding respondents above 6 or so is extremely low (see Zaltman et al., Journal of Market Research). That’s because fundamental wants and needs are intrinsic to people everywhere, and when you are looking at the structural or foundational aspects of what jobs people want to get done, you will discover fundamental truths.

The value of the 4-6 person deep dive is in understanding the total customer experience, the before, during and after of behavior, the decision-making calculus that drives choice and preference, and the subconscious short-cuts and faults that yield imperfect results and latent dissatisfaction – and where you’re find the raw material for true innovation.

We have much to learn from medical science, which has embraced precision medicine (following the mapping of the human genome) and is now developing early stage clinical trial designs that combine “n of 1” studies, going deep into the molecular biology of an individual first, in order to develop subsequent interventions for large populations. The same “inverse sample” principle holds true for innovation: Isolate and discover fundamental truths and deeply held beliefs and barriers in the individual first, rather than in the messy, heterogeneous, large-scale population.

The final guidance on “small n, Big P” is to find the right 4-6 people for your discovery. Look for people who are:

Self-efficacious: they believe they can improve their lives and seek to do so (they’re not lemmings or risk-averse laggards);

Good at implicit association: they make connections across diverse subjects, so you can explore all kinds of alternate realities, future scenarios, what ifs and actually talk about dreams;

Hyper-articulate: they can give voice to difficult thoughts, turn feelings into words, access their subconscious if given the opportunity to do so.

Remember, your goal is to learn fundamental truths, not to build a test market population!

Finally, make half your sample Maximizers, — who seek to optimize their options and are driven to compare choices, make corrections, edit their thoughts and actions, strive for maximum gain – and make the other half Satisficers — they are pragmatic, don’t dwell on regrets and will help you to find exactly where to “cross the chasm.”

Rule #2: “Invest more in Upstream than Downstream”

As a simple corollary to Rule #1, insights come from early, exploratory customer discovery work, where all options are on the table and the search for insights is not constrained by received wisdom.  The NPV of insight acquisition in a defined and mature market space is virtually nil; new markets, new rules, new solutions require a very different approach (remember Steve Jobs’ admonition).

The historic ratio in market research between Upstream/Discovery and Downstream/Validation is about 1:4 in Fortune 500 companies. Much of that derives from the use of market research as a decision support tool, rather than as a means to unlock growth opportunities or to drive innovation.  You can probably reduce up to half of the current Downstream work just by eliminating waste and redundancy and then apply the proceeds towards Upstream.  Drop the CYA studies and you will be fairly well funded for innovation.

Essentially, you should tie your investment in market research to your business portfolio and spend to opportunity, i.e., invest proportionately in growth and innovation. If you want to develop new market spaces, or if you participate in a highly dynamic sector that is subject to new entrants and disruptions, then you should be spending more in Upstream than Downstream.  If you are unhappy with your return on innovation, then look more to Upstream research to help and put the current research output in a lock box.

The revolution in customer discovery driven by the Lean Start-Up Movement, and by the thought leadership of Steve Blank, Eric Ries and other rebels, emphatically demonstrates that early stage customer encounters, with Minimally Viable Products and “what if?” scenarios, gives innovators the ability to iterate new opportunities collaboratively with customers and to rapidly pursue a real process of deep discovery, managing risks and maximizing rewards all the way. Upstream customer discovery gets at fundamental truths and then surrounds them with deep probing and constant validation in order to find actionable pathways to innovation.

Rule # 3: “It’s all about Problem Definition”

When it comes to innovation, just as in the scientific method, the key to understanding customers’ unmet wants and needs are comes down to problem definition. Define what problems your customers are trying to solve differently and the innovation possibilities grow exponentially.

The breakthrough that came with the “jobs to be done” (JTBD) notion pioneered by Clay Christensen has had a positive impact on companies’ ability to think about customer behavior differently. Essentially, JTBD reframes the problems that customers face and the ways in which they try to solve these problems – “what does the customer want to accomplish?” (not “what does the customer want to buy?”), “what is the full landscape of customer choice?” “what performance measures do customers use?” “who would the customer hire to perform this job?”

JTBD has opened the aperture to reveal a wider set of alternatives to current market structure and customer behavior, and that reframing of the problem has forced companies to think about how relevant and differentiated their value proposition really is.  Does the customer really want a milkshake, or a way to make a long commute more tolerable? Does the Starbucks user want overpriced, derivative coffee, or free WiFi and a place to hang out? Does the Twitter user want news and information or an alternative to Angry Birds? Does the prescription drug user want a way to stay compliant with a course of therapy or a way to alleviate the burden of dealing with a chronic condition?

This new approach to Problem Definition with customers is a good thing, but it becomes even better if the Customer JTBD is linked with a Technical JTBD. If the Customer JTBD is “I don’t have a car and I need to get from place A to place B” then Technical JTBDs might be a rental car (Avis), a shared rental car (Zip Car) or a shared ride service (Uber), with the geo-location service and mobile app and pricing model to go with the solution of choice.

This linkage of Customer JTBD with Technical JTBD is advanced problem definition, allowing entrepreneurial types to explore all kinds of innovative solutions, outside the current frame of reference, as part of an iterative process of customer discovery. By linking Customer and Technical JTBD, you create more areas for exploration because there are two points of tension to resolve, and therefore more combinations, more continuous redefinition of the problem, learning that builds and feeds off itself to push the hypotheses forward. All in all, a terrific way to find new terrain.

Rule #4: “Mind, Body, Soul, Task”

All customers are good at deluding themselves that they are acting rationally and in maximizing self-interest. They are clever and consistent in denying their “System 2” instincts that the Nobel laureate Daniel Kahneman observed, the fast, unconscious and often erroneous decision-making process built on tricky rules of thumb and ego depletion. And, perhaps surprisingly, business decision-makers might be even more irrational than consumers. The appearance of economic maximization is a false positive most of the time. When the outcomes of most businesses are full of risk and uncertainty, expect anything but rational decision-making.

Innovation requires powerful insights that generate new concepts. This requires discovering what lies in the subconscious — core motivations, deep meaning, hopes, fears and dreams. This crucial difference requires methods and techniques that are entirely different than those used in traditional market research.

Consumer wants and needs and real world JTBDs can only be elicited through deep discovery. Take the time to go deep into the psychology and decision making of your “small n” group to understand their total experience — from rational to emotional to subconscious decision making, through actual behaviors.

The objective is to capture the articulated and unarticulated experience, needs and dreams of your target customer—in order to determine how best to develop innovative products and solutions to deliver on their needs.

Looking at the same ideas through different lenses—Mind, Body, Soul and Task— reveals patterns or incongruities that address customers’ hidden truths. Mind is the rational, self-stated, ostensibly objective area of choice and preference that our optimistic and overconfident selves crave. We know what we want and we know it when we see it. Body is the inverse of Mind, the purely sensorial and visceral world of feelings, not thoughts, hard to express but hugely more important – after all, the brain is built around sensory impulses, not analytic thinking. Soul is the place that we preferentially exclude from everyday consciousness, the hidden fears and low expectations that we really experience, but that are too daunting or too painful to dwell on most of the time. Task is the imperfect and compromised way in which we set out to rationalize the gaps between Mind, Body and Soul – accepting the status quo, settling for unsatisfactory outcomes, trading off requirements that we would really like to see delivered, buying goods and services that we feel we have to, rather than want to.

Mind, Body, Soul and Task can all be discovered through deep dives with a set of 4-6 customers through a series of elegant protocols that have emerged from cognitive science. We can often figure out pretty closely what people need but cannot articulate. We can map the total customer experience, and the rational and subjective states of mind, and find those platinum-coated nuggets of real insight, revealed in the gaps between what people say they want vs. what they really need, what they say they do vs. how they really act, and what they dream about versus what the settle for.

This isn’t market research that is too big in sample size for insights, or looking to confirm existing beliefs, or gullible and willing to believe everything that people say.

For sure, unmet needs are very elusive. Consumers cannot readily tell you about what they don’t have, and they struggle to imagine different results. But if you apply the lenses of Mind, Body, Soul and Task, you can identify deeper truths. By digging deep, and then looking for patterns or incongruities, we get closer to the consumer’s true hopes and fears.

Rule #5: “Embrace Big Data”
A final driver in the revolution in customer insights is the arrival of Big Data.

Big Data, and the huge, rapid processing and analytical capabilities that go with it, enable companies to understand customers in new and unexpected ways. We can now analyze huge, multivariate data sets – often publicly available – and discover patterns that could not be observed previously. In the past, companies looked for causality – understanding why sample populations behaved in certain ways – but Big Data shows us that correlations are much more important than causality. The empirical evidence derived from processing Big Data finds multiple factors that are correlated, even when the data themselves appear random or independent of one another.

Big Data is showing us that it’s no longer quite as important to hypothesize why something happens as much as it is to understand when it happens and how it happens. Big Data teaches us to mine as much data as possible and then first to “let the data speak to us,” rather than to hypothesize meaning and causality from smaller data sets, which can often lead to confirmation bias and limit discovery of unexpected patterns.

So, it turns out that there is a place for quantitative digging in the upstream customer discovery process, but this place is for identifying new opportunities, not confirming existing beliefs.

Surprising and disruptive innovations that are coming from insights born of Big Dta include: the assessment of individual health risk for employers by profiling hobbies, media consumption, web shopping, personal messaging, etc.; smart cars that can prompt drivers to avoid accidents, by analyzing body movement; self-cleaning engines, optimized for peak performance, discovered by crunching through years of maintenance data; real time collection and indexing of customer service information to reduce errors, fix fleets and eliminate service calls.

The best thing of all about Big Data is that it levels the playing field and allows the small company or the solo entrepreneur to compete with the bigger company or the industry giant. Open source software such as Spark creates a 100x improvement in the speed of the data analytics process. Literally hundreds of thousands of valuable data sets on demographics, industry and technical performance are available for free. Innovators can now supplement their own hypotheses with a whole new set of potentially game-changing pathways, simply by letting the data speak.

The revolution in customer insights is moving, as all revolutions do, in fast and unpredictable ways. The only thing for sure is that the company that sits on its knowledge base and sticks to the traditional approaches to insight generation is bound to be disrupted by those who heed the call and embrace the new world of small n, upstream discovery, advanced problem definition, Mind-Body-Soul-Task and Big Data.

Tim Munoz

Managing Director

Share