Monday, November 19, 2012

Measurement Bias Destroys Products

Flying cars have always been a staple of sci-fi. And why not, just think how fun it would be to drive one!

Suppose then, that flying cars are on the horizon. All the futurists, and you, agree that they are going to completely replace existing cars and car design. There've already been field tests and the media is gobbling up the latest info - especially when they are available to buy! Now, you run a business which creates & sells cars. How do you go about introducing a flying car that your existing and future customers will want?

You decide to go with perhaps one of the best tools available to you - talking with your customers! Yes, yes of course you do this! We talk with our customers to learn about their wants and needs and then we solve those needs!

Do they not know, or not care?

You begin talking with your customers and something odd happens... nobody seems interested in having their cars fly! They keep talking about improvements on existing designs - better gas milage, better brakes, a smoother ride on the road, four wheel drive.....lots of things except flying. Now, they must know about flying cars because it’s all over the news, but still, your data is telling you they don’t want it? What’s going on?

Data data everywhere, and not a sample to trust!

The product idea: ‘people don’t know what they want until you show it to them’, can be an issue but it’s not always the case. The problem may be with your data. In statistics there exists the phenomenon of measurement bias. Some of the most notable are:
To sum all of these up into an example, remember our flying car situation. You know flying cars are going to be the future, but your data says that your customers just don’t express an interest. Instead, your data (customers) tell you to improve existing features. When you talked with your customers, let’s think about the responses you got and the responses you didn’t get.

When you approached your customers for feedback, the ones who answered were likely ones who:

  • Were willing to take the time to talk to you

  • Really like your product the way it is

  • See products in the context of what they want / need now instead of what they will need in the future. (present oriented vs future oriented)

What about the customers you didn’t reach (or got false answers). That might be because:

  • Social desirability bias

  • Your customers are busy (let’s assume b/c they value their time - an important trait among successful people)

  • Are neither particularly satisfied nor unsatisfied with your product

  • Customers of other products or those who are not yet your customers

Sure, the 1st group will give you great feedback on what your product should be now...but what about in the future? What about when those improvements the first group told you to work on end up being obsolete when you’re done?

In reality, the 2nd group is by far the most interesting for you. Successful customers are more willing to pay for products which continue to help make them successful and those who are ambivalent about your product are more likely to switch to a competitor. Reaching these customers, while also working around Social desirability bias is the key to developing your product.

Too much attention to the wrong group can destroy your product. In a world of flying cars, don’t be still tweaking your 4 wheel drive for a handful of vocal customers because you weren’t able to reach the silent majority; otherwise, the only indication you ever had a silent majority is when you notice most of your customers had already left.