June 13, 2019

Prioritizing product experiments using unit economics

How does your team prioritize what you work on? What sort of process do you have? Where do you feel like it’s falling short?

Every product team out there has some system for determining what they’ll work on next. Some teams have a single decision maker, others have consensus-driven order, and some use a codified framework.

Over the past year and a half, the Drift growth team has tried it all. As we’ve gone through different stages of company growth and product development, our growth team has experimented with a variety of different systems for determining our next move.

Through that experimentation and trial process—along with a bit of inspiration from our friends at Dropbox—we unlocked a system of prioritization that helps us align every product decision we make with the rest of our business.

Popular prioritization frameworks

Before we get into our own framework, let’s review some of the more popular prioritization methods out there.

Many of the most popular systems for prioritizing product experiments use some derivative of the ICE framework:

  • Impact—How impactful will this experiment be?
  • Confidence—How confident are we that this experiment will validate our hypothesis?
  • Effort—How easy is it to run this experiment?

A score is assigned to each variable (e.g. impact = 7 out of 10, etc). Those scores are then added up and the projects with the highest scores are prioritized. At a high level, all of the product teams I’ve talked with use some variation of this system to determine what they should work on next.

After chatting with Darius Contractor—who helped run growth at Dropbox for 4 years and is now Head of Growth at Facebook Messenger—the Drift team was inspired by a system that Darius called EVELYN. Part of what makes his system so interesting is that it allows teams to manage an expected pipeline of revenue impact through something called opportunity sizing.

Thibault Imbert at Adobe has a similar system, which takes a variety of inputs to determine the possible output of any product or funnel change in dollar terms.

Basically, on top of figuring out the confidence, effort, and impact, you also spend some time to understand the actual numbers behind that change. For example, ask yourself: If we were to make this change, based on how many people use the product each day, how far can we realistically move that number?

Prioritization through unit economics

What made Darius’ and Thibault’s frameworks especially effective was that they were both tied to revenue. We wanted to take this further and boil product experiment prioritization down to its most objective core.  

And we found that the best way to do this was to figure out the unit economics for each of our levers. So, for example, we wanted to be able to say:

  • A Product Qualified Lead (PQL) = $2
  • An active free user = $1
  • A self-service purchase = $100
  • A sales demo booked = $20

We dug into our company’s funnel and identified roughly 20 different levers that we could impact by making changes to the product or the systems generating leads (product changes, onboarding, website, prospecting data, etc).

From there, we settled on the values that we felt were most important for determining what our team should work on next. These ended up being:

  1. Hypothesis (with a metric and a timeframe)
  2. Impact metric (PQL, active user, etc)
  3. Opportunity size (how many impact metric units could we drive if this worked?)
  4. Confidence %
  5. Estimated # of days

In Airtable (my new favorite organization tool of choice), we then crafted a formula that could give us a value per day metric:

(Impact metric  * Opportunity size per month * Confidence) ÷ Estimated days to build

The table would then be sorted by this value per day field, giving us a very clear list of things to work on, in order of highest expected financial value.

this is a screenshot of an Airtable field for expected value per day. This is an example of a system for prioritzing product experiments u

An objective system of prioritization

Implementing this system allowed our team to avoid debating what we each felt was important. Instead, we now found ourselves debating the validity of the data around the inputs. This removed both emotion and louder voices from the discussion around prioritization and enabled us to become more closely aligned around business goals and revenue.

Ultimately, this method of product prioritization gave everyone on our team a very clear view of how the work they do impacts the business’ bottom line, helping us become better aligned with the rest of the company.

Matt Bilotti is Product lead, Consumer at AngelList. Prior to AngelList, he led Growth at Drift, where his team owns product-led acquisition and activation for Drift’s free product lines. He’s been with the company since they signed on their first customers and had many hats along the way. Before Drift, he spent time at HubSpot and other Boston-based startups doing product-related work.

Learn more