How do you feel when the success of your design hinges on metrics? For me, I used to not like it. I felt that it was too rigid and that design shouldn’t be reduced to a number game, which I talked about earlier. Since then, I’d been trying to learn the best practices of integrating data into design, and this book Designing with Data — written by Rochelle King, Elizabeth F. Churchill and Caitlin Tan— helped me understand better the role data plays in product design. I’m sure you’ll benefit a lot from this book, just like I did. Below are my learnings.
Data-Driven < Data-Informed < Data-Aware
We often come across the term data-driven design, but we might not know that there are different ways to leverage data in design, and data-driven is only one way. This book provides clear definitions of three different approaches: data-driven, data-informed, and data-aware.
Data-driven means that design decisions are determined by data. This approach is best taken when both the problem and the goal have been clear and unambiguous. Data-informed, compared to data-driven, is less rigid and more expansive. We haven’t fully grasped the problem yet, and so we just want to let data inform and inspire the way we think about and approach the problem. Finally, data-aware is about noticing the different types of data available and choosing the right types to help identify problems and define success. It’s even broader than data-informed.
Having this language to describe the relationship between data and design is crucial. As a product designer, you may know the team is looking at data to make a shipping decision, but you may not know whether data is being used in the best possible way if you are not familiar with these concepts. Let’s say we haven’t fully explored the problem space but the team has decided to ship a design simply because the data is not bad and stopped investigating. This could imply that we’ve missed the opportunity to uncover other possibly even better solutions.
Another concept introduced in the book that I think is also really important is this idea of local vs global and evaluatory vs exploratory. When the change in design is local, only one variable is altered and it’s easy for us to establish the causality relationship between design and metrics through an A/B test. When the change is global, many variables are altered simultaneously and multiple A/B tests may be needed in order for us to tease out the the different factors that may have affected the metrics. When we are evaluatory, we hone in on one or two designs, measure them, choose between them and optimize them. When we are exploratory, we think more broadly and take more risks. We open ourselves to different design possibilities and are eager to test out and learn from them, so we can come up with even better designs in the subsequent iterations. In this case, the designs we test may not even need to be polished.
Hone Design Instinct Through Data
When we put our design to an A/B test, we usually have our assumption, or hypothesis in mind: By making a particular change in design, we expect to see certain changes in user behaviors and metrics. Our hypothesis often comes from our design instinct or intuition. It may or may not be correct, and will be either validated or disproved by the performance of our design in the wild. The benefit of leveraging data in the design process is that over time it can hone our design instinct and intuition and helps us become better product designers. It also encourages everyone in the team to put their ego aside and look at things more objectively, especially when compared to experience-driven design or intuition-driven design. The book mentions how Reed Hastings — the CEO of Netflix — would submit his own ideas to the same rigor of A/B testing just like everybody else, and how product teams are placing bets on the outcomes of A/B tests to have a bit of healthy competition in product sense, which is fun.
The book also discusses the importance of having this mindset of designing to learn rather than designing to ship. Doing so prevents us from getting too hung up if things don’t turn out as we expected. Every test is an opportunity to learn something new or valuable, especially when it fails. This mindset allows us to see far and wide, so we won’t prematurely hone in on a particular outcome and we’ll be more likely to come up with an innovative or interesting solution.
Be Meticulous about Data Collection
Data, if we are not careful, can be misinterpreted and leads us to unsound conclusions. While designers are not data scientists, knowing the basic language related data collection and statistics can help us interpret the results of an A/B test in a proper way.
The book talks about the importance of achieving statistical significance and its closely related concept: the p-value. The p-value refers to the probability of observing the current results in a random manner. It’s important for the p-value to stay close to 0 so we can confidently say that the design changes we made are actually the reason for the changes in metrics. The length of an A/B test is another thing we should pay attention to, because users tend to behave differently at different times. That’s why many A/B tests last for a whole week or a whole business cycle to reduce biases.
Final Thoughts
So after reading the book, did I change my way of looking at data? Yes, definitely. Designing with data doesn’t have to be rigid and suffocating. The key is to understand when we should be data-driven, when we should be data-informed, and when we should be data-aware at which stage of a design project. And design won’t be reduced to numbers if we can have a holistic view of data. The book stresses the importance of leveraging different data types, not just quantitative data but also qualitative data from surveys and interviews, to triangulate the results, so we can more accurately gauge the performance of our design. Data is a powerful tool. When used properly, it can be insightful and liberating.
Recommendation: ⭐️ ⭐️ ⭐️ ⭐️ ⭐️
My Learnings from Other UX Books
- Be an Excellent Facilitator: Learnings from Reading Communicating the UX Vision
- Let Ideas Collide with Reality: Learnings from Reading That Will Never Work
- Learnings from Reading UX Strategy: How to Devise Innovative Digital Products that People Want
- Learnings from Reading About Face
- Learnings from Reading A Practical Guide to Information Architecture
