“Measure What Matters” 40 E-Commerce and Marketplace Product Metrics
How to define and measure the most useful success and guardrail metrics for a user-focused marketplace driven e-commerce product
Successful product companies know how to define product and feature metrics and how to iteratively optimize the product to improve these metrics. Before defining 40 marketplaces specific product metrics, I’ll paste this amazing story behind YouTube’s north star metric, Watch Time, and how it shaped the way YouTube was built. This story from the book “Measure what Matters” by John Doerr and has stayed with me for a long time and has in fact in many ways shaped the way I think about products.
Watch Time and Only Watch Time
Cristos Goodrow, Vice President Engineering at Google: In September 2011, I sent a provocative email to my boss and the YouTube leadership team. Subject line: “Watch time, and only watch time.” It was a call to rethink how we measured success: “All other things being equal, our goal is to increase [video] watch time.” For many folks at Google, it smacked of heresy. Google Search was designed as a switchboard to route you off the site and out to your best destination as quickly as possible. Maximizing watch time was antithetical to its purpose in life. Moreover, watch time would be negative for views, the critical metric for both users and creators. Last (but not least), to optimize for watch time would incur a significant money hit, at least at the start. Since YouTube ads were shown exclusively before videos started, fewer starts meant fewer ads. Fewer ads meant less revenue.
My argument was that Google and YouTube were different animals. To make the dichotomy as stark as possible, I made up a scenario: A user goes to YouTube and types the query “How do I tie a bow tie?” And we have two videos on the topic. The first is one minute long and teaches you very quickly and precisely how to tie a bow tie. The second is ten minutes long and is full of jokes and really entertaining, and at the end of it, you may or may not know how to tie a bow tie.
I’d ask my colleagues: Which video should be ranked as our first search result? For those at Google Search, the answer was easy: “The first one, of course. If people come to YouTube to tie a bow tie, we surely want to help them tie a bow tie.” And I’d say, “I want to show them the second video.”
And the Search cohort would protest, “Why would you do that? These poor people just want to tie their bow ties and get to their event!” (They were probably thinking: This guy’s insane.) But my point was that YouTube’s mission was fundamentally divergent. It’s fine for viewers to learn to tie bow ties, and if that’s all they want, they’ll choose the one-minute manual. But that’s not what YouTube was about, not really. Our job was to keep people engaged and hanging out with us. By definition, viewers are happier watching seven minutes of a ten-minute video (or even two minutes of a ten-minute video) than all of a one-minute video. And when they’re happier, we are, too.
It took six months, but I won the argument. On the Ides of March 2012, we launched a watch time-optimized version of our recommendation algorithm aimed at improving user engagement and satisfaction. Our new focus would make YouTube a more user-friendly platform, particularly for music, how-to videos, and entertainment and late-night comedy clips.
Our true currency wasn’t views or clicks — it was watch time. The logic was undeniable.
YouTube needed a new core metric.
Marketplace Metrics
Circling back to marketplaces, any product and feature hypotheses/ ideas should ideally increase many of the product metrics listed below and this can be validated with A/B (Randomized Controlled Trials) or multivariate or multi-armed bandit experiments. Experimentation will be critical to test hypotheses and assess the impact of ideas. A platform and infrastructure that democratizes experimentation, enabling everyone in the team to run experiments will have high returns and since the use case is quite generic, the platform can be used across other departments. Booking.com for instance has an experimentation platform that enables the team to run experiments via an SDK and through code. The large majority of the successful use cases of feature building in a commercial context have been enabled by sophisticated and iterative experiment designs, either to guide the strategies or in order to detect their impact.
Seller Metrics
1. Number of Sellers Available = Total number of sellers on the marketplace
2. Number of Products Sold = Total number of products sold by the sellers
3. Average Products per Seller = Total number of products available in the marketplace to the total number of sellers available
4. Average Sales per Seller = Total number of products sold by the sellers to the total number of sellers available on the marketplace
5. Monthly Active Sellers (MASs)= Number of sellers with at least 1 product sale in the past month
6. Weekly Active Sellers = Number of sellers with at least 1 product sale in the past week
7. Average Revenue per Seller = Total seller revenues to the total number of sellers in the marketplace
User Metrics
8. Conversion Rate = Total number of users buying products to the total number of users shopping in the marketplace
9. Total revenue for a period = Sum of revenue generated or transaction values in the time period
10. Average Revenue per User = Total revenue generated to the total number of users
11. Average Order Value or Average Transaction Value per Order = Total revenue generated to the total number of orders
12. Retention Rate for a Period = (Number of users at the end of the period — number of new users acquired in the period) to the number of users at the beginning of the period
13. Churn Rate for a Period = Number of users lost in the period to the number of users at the beginning of the period
14. Customer Lifetime Value = Total revenue generated by the user — costs incurred by the organization for user acquisition and retention
15. Lifetime Value = The total revenue generated by the user in a time period — the total costs incurred by the marketplace in both acquisition and retention of the user for the same period.
16. Monthly Active Users (MAUs) = Number of users with at least 1 purchase in the past month
17. Weekly Active Users (WAUs) = Number of users with at least 1 purchase in the past week
18. Product Stickiness = Weekly (or daily) active users to the monthly active users
19. No. of Products Available = Total number of products offered on the marketplace platform (not necessarily a metric to maximize)
20. No. of user sign-ups = Total number of users that signed up using an email or other sign in methods
21. No. of Weekly/ Monthly Orders = Total number of weekly orders and the total number of monthly orders
22. New Users Ratio = Total number of new users signed up in a time period to the total number of users in the platform
23. New Buyers Ratio = Total number of new buyers of products in a time period to the total number of shoppers in the platform
24. Lurkers Ratio = Total number of buyers to the total number of signed up users in the platform
26. No. of Products purchased= Total number of products purchased
27. Product Utilization Ratio = Total number of products purchased to the total number of products being sold at a given point in time
28. Recency for a period = Days elapsed since the user’s last purchase calculated at the end of the period
29. Frequency for period = Number of orders, users have placed in the time period
30. Engagement/ Addiction = Average time spent on the platform by all users per month/ week
31. Net Promoter Score = From the user research world, that typically takes the form of a single survey question asking users the likelihood that they would recommend Outschool to a friend or colleague.
Channel Metrics
32. Organic Users = Total number of users from free/ unpaid channels
33. Channel Conversion Rates
34. Device Conversion Rates from various devices like web, mobile, iOS, and android
35. Device User Count and Device Product Count
36. Channel ROI = Total revenue generated to the total costs incurred in acquisition and retention
37. Channel Portfolio to verify a slow shift to organic and direct channels
Search Metrics
38. Number of searches with results = Total number of searches that resulted in products/ results
39. Number of searches with no results = Total number of searches that resulted in no results
40. Number of high intent searches = Total number of searches with user-entered search keywords
I’ll go back to the YouTube story before ending this piece. I doubt YouTube could have scaled the heights it did without the process, structure, and clarity of that stretch metric. In a fast-growing company, it’s a challenge to get everybody to align and focus around the same objective. People need a benchmark to know how they’re performing against it. The catch is to find the right one. The billion hours of daily watch time gave the tech people at YouTube a North Star.
But nothing stays the same. In 2013, the watch-time metric was the best way to gauge the quality of the YouTube experience. But YouTube started looking at other variables, from web-added videos and photos to viewer satisfaction and a focus on social responsibility. If a user watches two videos for ten minutes apiece, the watch time is the same — but which one makes the user happier?
As early as 2015, YouTube began to advance beyond watch time by factoring user satisfaction into the recommended videos. By asking users about the content they found most satisfying, and measuring “likes” and “dislikes,” they could better ensure that they’d feel their time on YouTube was well spent.
