Reading Notes 2021 May - Jun

This is my third blog of this series, summarising the great posts Elise and I came across during our Friday and Sunday night reading sessions. In these two months, we have been reading a lot about how to define the right product metrics. And as we read more on this topic, Medium recommend more and more posts on this topic to us (you know why). Maybe we need to tweak our reading habits a bit… Hope you enjoy the reading as we do :)

Best of The Two Months

As every product has a north star metric, here is the north star blog for these two months :)

A Brief History of Netflix Personalization:
When I first saw the name, I thought it would talk about the evolution of the recommendation system at Netflix. However, this article talks about more than that – it covers product changes, features launches, model improvement, metrics improvement, and both succeeded and failed, and how they work together to improve personalization at Netflix. A very good reading of how a product iterates and succeeds.

Product Metrics

1. 7 Metrics to Help You Make Smarter Decisions During the Product-Market Fit Phase: User-centric KPIs for product-market fit
2. North Star Metric - Measure the Right Thing: Three tiers of product success metrics and how to define your North Star Metric
3. Should I be using NPS to measure customer loyalty in SaaS?: An interesting discussion of drawbacks of using NPS for SaaS products, and potential alternatives
4. Where eBay Went Right — and Wrong — with AI: What You Measure Matters: A real example of why a machine learning model with good performance failed to deliver financial result due to a bad choice of metric
5. Why We Use Experimentation Quality as the Main KPI for Our Experimentation Platform: A very good discussion of how to define the right metric for an experimentation platform and why
6. Measuring Impact — Picking the right metrics in Product: Important criteria to pick the right product metrics
7. Driver Trees — How and Why to use them to Improve your Business: Use KPI trees to decompose key metrics, align the goals, and improve business
8. Product Metrics That Matter: Talks about the AARRR and HEART metrics framework and common metrics under them

Customer Experience, User Research, and Analytics

1. How to Measure Customer Experience: Brief introduction of customer journey map and the a three-step approach to measure the customer journey
2. The Beginner’s Guide to Creating a Customer Journey map: Key elements of a customer journey map
3. How to Set Up Your Product Analytics: Some good practices to instrument product analytics tracking
4. How to Start Planning User Research and Analytics for Your Product: Analytics framework to collect user feedback

Experimentation and Causal Inference

1. Geo Experiments, The Perfect Complement to A/B Testing: When and why to use geo experiments and the limitations
2. A New Method to Measure the Results of your A/B Tests: How we should define ‘test win rate’ and evaluate all the tests we ran

Machine Learning

1. The Intuition behind Reinforcement Learning: Explanation of the terminologies in Reinforcement Learning
2. xgbse: Improving XGBoost for Survival Analysis: A survival analysis package built on top of xgboost
3. 5 Outlier Detection Techniques that every “Data Enthusiast” Must Know: A clear explanation of 5 common outlier detection methods

Others

1. Data-Driven Marketing Attribution: A data-driven attribution model based on the Shapley value concept taken from cooperative game theory
2. Markov Chain in Multi-Channel Attribution Modeling: How to use Markov Chain to do multi-channel attribution
3. 3 Visualizations that Changed My Life: Three special visualization types that help to present data stories better
4. How does Airbnb Track and Measure Growth Marketing?: Introduces how Airbnb tracks marketing campaigns for growth marketing
5. How Airbnb Achieved Metric Consistency at Scale: How Airbnb restructured their metric platform to achieve metric consistency