The Underappreciated Role of the Consumer in Innovation

After my previous post, I was prepared to like The Myths of Innovation by Scott Berkun.  I did like it, but my enthusiasm for the book was tempered by Berkun's treatment of "the best ideas win" myth.  I believe his observation is correct: the relationship between the subjective quality of a prospective innovation and the rate and extent of its adoption is ambiguous.  However, I suspect that Berkun may underestimate the crucial role played by interdependent decision-making among consumers.

Early in his book, Berkun acknowledges how adoption by consumers is a socially contingent process:

The love of new ideas is a myth: we prefer ideas only after others have tested them.  We confuse truly new ideas with good ideas that have already been proven, which just happen to be new to us.

But, even as he acknowledges the "evolutionary advantage in this fear of new things," Berkun seems to consign consumers to a secondary role in the process of innovation:

Even early adopters, people who thrive on using the latest things, are at best adventurous consumers, not creators.  They rarely take the same risks on unproven ideas as the innovators themselves.

In fact, he seems to suggest that consumers timidity and inertia slows the rate of adoption of the best ideas.  That is, Berkun posits that there is an inverse relationship between the rate of adoption and the barriers to adoption:

The reason why Internet and cell phone usage climbed faster than previous technologies isn't because things happen faster today...It's simply because the barriers of entry were low.

Furthermore, he suggests that the "goodness" of an idea is inversely related to the ease of adoption (see Figure 8-2):  "This suggests that the most successful innovations are not the most valuable or the best ideas, but the ones that appear on the sweet spot between what's good from the expert's perspective, and what can be easily adopted, given the uncertainties of all the secondary factors combined.  The idealism of goodness and the notion that goodness wins is tempered by the limits and irrationalities of people's willingness to try new things, the culture of the era, and the events of the time."  So, as I understand him, Berkun argues that there is a causal relationship between the net benefit of an invention and the speed and extent of adoption.

It's a plausible and intuitive argument.  Doug Hall, in his marketing physics framework, makes a similar case.  Hall concludes that a clear articulation of overt benefit ("What's in it for the customer?"), real reason to believe ("Why should the customer believe you will deliver on the promise?"), and dramatic difference ("How revolutionary and new-to-the-world is your benefit/reason to believe pair?) drives adoption.  Importantly though, Hall acknowledges that there is not a linear cause-and-effect relationship: having a great concept means that the chances for widespread and sustained adoption can be increased to over 50%.

So, if the quality of an invention might explain 50% of success, what explains the other 50%?  Recent research suggests that the dynamics of the inter-relationships among consumers is a very likely candidate.  Consider the following, from a recent New York Times article by sociologist Duncan Watts:

Conventional marketing wisdom holds that predicting success in cultural markets is mostly a matter of anticipating the preferences of the millions of individual people who participate in them...The common-sense view, however, makes a big assumption: that when people make decisions about what they like, they do so independently of one another.  But people almost never make decisions independently--in part because the world abounds with so many choices that we have little hope of ever finding what we want on our own; in part because we are never really sure what we want anyway; and in part because what we often want is not so much to experience the "best" of everything as it is to experience the same things as other people and thereby also experience the benefits of sharing...

Yet our mutual dependence has unexpected consequences, one of which is that if people do not make decisions independently--if even in part they like things because other people like them--then predicting hits is not only difficult but actually impossible, no matter how much you know about individual tastes...The reason is that when people tend to like what other people like, differences in popularity are subject to what is called "cumulative advantage," or the "rich get richer" effect.  This means that if one object happens to be slightly more popular than another at just the right point, it will tend to become more popular still.  As a result, even tiny, random fluctuations can blow up, generating potentially enormous long-run differences among even indistinguishable competitors...

Watts goes on to describe a really interesting experiment that measured the popularity of products (songs) in independent and socially influenced conditions:

This setup let us test the possibility of prediction in two very direct ways.  First, if people know what they like regardless of what they think other people like, the most successful songs should draw about the same amount of the total market share in both the independent and social-influence conditions--that is, hits shouldn't be any bigger just because the people downloading them know what other people downloaded.  And second, the very same songs--the "best" ones--should become hits in all social-influence worlds.

What we found, however, was exactly the opposite.  In all the social-influence worlds, the most popular songs were much more popular (and the least popular songs were less popular) than in the independent condition...Introducing social influence into human decision-making, in other words, didn't just make the hits bigger; it also made them more unpredictable.

Watts also found that, while the quality of the product mattered, quality did not cause popularity in a straightforward manner:

So does a listener's own independent reaction to a song count for anything?  In fact, intrinsic "quality," which we measured in terms of a song's popularity in the independent condition, did help to explain success in the social-influence condition..."good" songs had higher market shares, on average, than "bad" ones...Overall, a song in the Top 5 in terms of quality had only a 50 percent chance of finishing in the Top 5 of success...social influence played as large a role in determining the market share of successful songs as differences in quality.

Here is what Watts concluded:

Our desire to believe in an orderly universe leads us to interpret the uncertainty we feel about the future as nothing but a consequence of our current state of ignorance, to be dispelled by greater knowledge or better analysis.  But even a modest amount of randomness can play havoc with our intuitions...That doesn't mean we should stop trying to anticipate the future, any more than we should stop trying to make sense of the past.  But it does mean that we should treat both the predictions and explanations we are served...with the skepticism they deserve.

So, read The Myths of Innovation.  Berkun's chapters on the myths of epiphany and the lone inventor are worth the price.  But be wary of the author's possible diminishment of the role of the consumer in the process of innovation, and be skeptical about his post hoc narratives regarding the success or failure of inventions.  Berkun's rationalizations are plausible and understandable, but they distract from the distinct possibility that the process of innovation is profoundly uncertain and luck plays a bigger role than we might wish.  Curiously enough, he seemed to acknowledge as much in the interview, about which I blogged previously:

Many stupid ideas have been successful and many great ideas have died on the vine and that's because success hinges on factors outside of our control.

On balance, I'd prefer to put my time and money behind a great idea.  In fact, I'd rather put my money behind three or four great ideas.  That way, I'll have a 90% chance of at least one success, if Watts and Hall's research is correct.  As long as I can keep the cost of trying down, I'll have a decent chance of surviving long enough in the marketplace to keep playing the game.

For more about cumulative advantage, check out The Social Atom, Critical Mass, and Six Degrees.  Also, check out the paper on venturesome consumption by Amar Bhide.