Using Data to Find Growth Levers
The holy grail of analytics impact
In the previous newsletter we talked about how analytics can make a massive impact to a company’s bottom line by finding key leverage points (aka constraints).
A constraint is a point of maximal leverage in a system, such that when solved unlocks massive incremental value for the entire system.
This is a good place to start but we can do more, much more with analytics. We can discover causal relationships or strong, directional correlational relationships that influence or drive key metrics.
I recently read an article where the DuoLingo team reignited user growth to the tune of 350% so I decided to turn it into a case study of how to use data to find growth levers in your business. I’ll put the article link at the end.
The problem
DuoLingo is an app that uses gamification to help millions of people learn a new language. Since the company is still a startup, sustained user growth (preferably exponential) is a key indicator of its success. It’s something investors look for when they evaluate any company.
DuoLingo had grown tremendously in the past due to its game-like qualities and support for 40+ languages. In mid-2018 however that growth was slowing down and the product team was tasked with finding new ways to grow one of their key metrics DAU (daily active users) which meant increasing user retention.
On the surface this appears to be a problem tailor-made for applied analytics, but their first attempts fell flat.
First attempts at solving the problem
The traditional way this problem is handled is through what I call the “bets and best practices” framework.
Essentially you frame the problem as an “optimization problem” where you decide that DAU is your objective variable and now you need to brainstorm solutions and choose the best one.
Usually this resolves to looking around and seeing what everyone else is doing (best practices) and then choosing one and trying it out (bets).
For DuoLingo they tried two such approaches:
Increasing gamification
Adding referrals
This approach is fine but often there’s a key element missing: understanding why.
First DuoLingo team tried to copy some gamification patterns from a game they liked called Gardenscapes.
We prioritized working on retention over new-user acquisition because all of our new-user acquisition was organic, and, at the time, we didn’t have an obvious lever to pull to supercharge that. Also, some of us had a suspicion that we could improve retention through gamification.
Applying a simple bets and best practices framework, they borrowed a feature without understanding why it worked for the game and how it applied to DuoLingo.
Unsurprisingly it didn’t work.
Depressingly, the result of all that effort was completely neutral. No change to our retention. No increase in DAU. We hardly got any user feedback at all. I was deflated. The greatest effect the initiative had was on our team
After regrouping, their next attempt was to borrow another feature from another app: Uber’s referral feature. Again they applied bets and best practices, by simply building the feature and seeing what happened.
Results were the same.
We implemented the feature and hoped our second attempt would be more successful. Instead, new users increased by only 3%. It was positive, but not the type of breakthrough we needed. Still, the team doubled down and pushed through, shipping iterations to the referral program and making some other bets, but no avail.
Understanding DuoLingo’s retention process
I’ve learned the hard way that only when you really understand why something happens can you not only solve the problem but also explain it in simple terms.
With two back to back failures, the DuoLingo team had to reassess and that’s when they found some key insights.
First why gamification that worked for Gardenscapes didn’t work for them.
In hindsight, it became clear why the Gardenscapes moves counter was not a good fit for our product. When you are playing Gardenscapes, each move feels like a strategic decision, because you have to outmaneuver dynamic obstacles to find a path to victory. But strategic decision-making isn’t required to complete a Duolingo lesson—you mostly either know the answer to a question or you don’t
Then why referrals that worked for Uber didn’t work for them.
It also did not take long to understand why our referral program did not produce Uber-like success. Referrals work for Uber because riders are paying for rides on a never-ending pay-as-you-go system. A free ride is a constant incentive. For Duolingo, we were trying to incentivize users by offering a free month of Super Duolingo. However, our best and most active users already had Super Duolingo
Finally they built a model of how retention actually worked. They segmented users based on their engagement level into buckets and build a retention model.
With the model created, we started taking daily snapshots of data to create a history of how all of these user buckets and retention rates had evolved on a day-by-day basis over the past several years. With this data, we could create a forward-looking model and then perform a sensitivity analysis to predict which levers would have the biggest impact on DAU growth. We ran a simulation for each rate, where we moved a single rate 2% every quarter for three years, holding all the other rates constant.
This model allowed them to narrow down their focus on the blue box above (Current Users) which conveniently has an arrow looping back into itself creating a compounding effect. Current Users thus became their goal metric.
Finding growth levers
With the goal metric in sight, the team turned their attention to finding levers that could move CURR in a positive direction. They looked in the past to see if they had done anything to affect CURR but didn’t find much.
Now they could intelligently look at potential features and take more well-thought-out bets. The first was leaderboards.
This time, we would be more methodical and intelligent about features we added or borrowed. We made sure to apply the lessons from our prior efforts with gamification. After some consideration, we decided to bet on leaderboards.
And what do you know, much success!
The leaderboards feature had a huge and almost immediate impact on our metrics. Overall learning time increased by 17%, and the number of highly engaged learners (users who spend at least 1 hour a day for 5 days a week) tripled
Next they looked at notifications, being careful not to burn people out.
we decided to give the team a lot of freedom to optimize on dimensions like timing, templates, images, copy, localization, etc., but they could not increase the quantity of notifications without strong justification and CEO approval.
While that wasn’t immediate, over time it worked
Over time, through countless iterations, A/B testing, and a bandit algorithm, the team was able to generate dozens of small- and medium-size wins that have amounted to substantial gains in DAU year after year.
Finally they looked into streaks and found an interesting insight.
In the search for even more growth vectors, the APM on the Retention Team started exploring whether there was a strong correlation between retention and usage of particular Duolingo features. He discovered that if a user reached a 10-day streak, their chances of dropping off were reduced substantially.
Insights like these are possible for every data team if empowered to research and explore on their own. This is Tier 2 value which I covered in this post
I encourage you to read the entire post which I have linked here.
Until next time.