The Fallacy of Data-Driven Product Design

Alpha House and House of Cards — House of Cards vs Alpha House

In 2013, not one, but two tech companies took the leap into data-driven political dramas. Only, Amazon’s got an IMDb got a 7.6 rating, and Netflix’s creation got a 9.0. Both are well-funded companies, familiar with techniques such as iterative processes, collecting user feedback, and specialized data analysis. The difference lies in their treatment of data.

In this post, we’ll be comparing the difference between data-driven and data-informed product design.

It’s not about being data-driven. It’s about being data-informed.

Continuing on the Amazon and Netflix comparison, let’s have a quick look at their methods for developing their content on a similar theme. Alpha House’s team used hard data to make decisions. They rolled out a competition of drama series and invited viewers to watch sample shows. They collected data on user behavior (such as playback, pause, exits etc.) and data points on favorite elements in the shows. Once they understood user behaviour, they chose a show that checked all the boxes: Alpha House.

Netflix also analyzed a lot of data. They researched the top trending topics, stories, cast actors, crew, and show elements. However, after informing themselves on what made audiences tick, their creative team created a full product that didn’t require multiple tweaks: House of Card. Netflix was not trying to find the magic equation that would determine a hit show. They digested the data as a whole, took a step back, and kept it at the back of their mind in their creative process.

Netflix adopts the data-informed approach, which is hypothesis-driven rather than algorithm-driven. By having a hypothesis, the creative team strives to understand the big picture and sets out to verify, rather than proposing a silver bullet solution.

Don’t be driven by data.

Data is the answer. Without knowing the question, the data is meaningless.

Put another way, one needs to come at data from the right direction to know where to go with it. Data is now on most people’s radar because of concepts such as Lean Startup Methodology and A/B testing, which are built on the fundamental principle that we can test hypotheses, verify them, and create better products in an informed way. Many people know to collect data, but once it is collected, it must be interpreted. This means, rather than just following the data to the next step (i.e. “That’s what the data indicate! We should do that!”), it is important to pause and think about what the data means, and what it may not mean. Data-informed decision making is a framework, not a golden formula for success.

So, what should you do with the data?

The data is only a proxy.

“If I had asked people what they wanted, they would have said faster horses.” – Henry Ford.

Data is not king. It is a tool. As in the case of Henry Ford, there is sometimes a limit to what a user can tell you. Sometimes a customer doesn’t know what they want until they have tried it. The key, of course, is figuring out whether they would want it before they try it. The creative process is just that. It incorporates the information that data suggests, to create something that a customer cares about. Numbers are a proxy for preferences, but they don’t actually tell us what a customer cares about. For that, our gut instinct and human empathy.

Fortunately, customer empathy can also be collected. It can be built up through listening to your customers. Customer empathy provides the ‘why‘ for the data you obtain. You can collect that by doing some of the things below:

Conduct a survey
Interview your customers
Read their feedback
Launch a usability test
Whatever means to obtain feedback

As Julie Zhuo, Facebook Product Design Director put it:

“Data and A/B test are valuable allies, and they help us understand and grow and optimize, but they’re not a replacement for clear-headed, strong decision-making. Don’t become dependent on their allure. Sometimes, a little instinct goes a long way.”

It is safe to continually launch tests to optimize your products. However, optimization can only go as far, meaning you’ve reached the local maximum.

This indicates that you’ve hit the limit of the current product design. It also means you don’t have enough data to understand how to get to the best outcome in the biggest market. At some point, there needs to be an overhaul, and there is no data to safely fall back on for guidance.

One’s gut feeling is still important for product design. Gut reaction is not random, as Daniel Kahneman showed in his book Thinking Fast and Slow. It is based off patterns that our subconsciousness recognizes faster than our conscious mind. When Netflix collected the data, they synthesized it into their existing gut repository as they thought about the big picture of developing a new series. A good gut instinct can be honed with exposure to many different ideas, careful observation, and empathy towards others.

Use data wisely to back your intuition and brave ideas

Data is playing an increasingly important role in every aspect of business, media, and the startup world. It can challenge your assumptions, provide insights, and point you in a direction you had not known about. It can help you learn more about your user patterns so that you can design something that checks off their criteria. However, at the end of the day, users and viewers are not holding a checklist when they decide what they love. They do it on gut instinct, and so should you (once in a while).

If you’ve been thinking about a product for a while, and know your target demographic well, why not take a leap and try it out? Just make sure you think of it as an iterative process. You can read more about that in our A/B testing posts!

Reference:

WernickeHow to use data to make a hit TV show – Sebastian Wernicke

Book: Lean Analytics – Alistair Croll and Benjamin Yoskovitz