The Metrics That Actually Matter: A Field Guide to Avoiding Vanity Metrics
Your dashboard looks impressive. Your charts are colorful. Your CEO is nodding along in the weekly review.
But here's the uncomfortable question: do these numbers actually tell you if your product is working?
Most teams measure what's easy to measure, not what's meaningful. They optimize for metrics that look good in board decks but fail to predict business outcomes. This isn't just inefficient — it actively misleads decision-making.
Let's fix that.
Vanity metrics share three characteristics:
1. They always go up (given enough marketing budget)
2. They're easy to game without improving the product
3. They don't predict revenue or retention
"Total signups" is the classic example. It feels concrete. It trends upward. But if 90% of those signups never return, the number is meaningless.
Similarly, "page views" and "downloads" are metrics of attention, not of value delivered. A user who visits your site 50 times but never converts is worth less than one who visits twice and subscribes.
Before any metric goes on your dashboard, ask:
1. If this number goes up, does my business improve?
2. If this number goes down, can I act on that information?
3. Can someone game this without creating real value?
If you can't answer "yes" to the first two, or "no" to the third, find a different metric.
Replace: Monthly Active Users (MAU)
With: Weekly Active Users who completed a core action
MAU obscures churn. A user who logs in once a month isn't engaged — they're evaluating whether to cancel. Instead, track users who perform the action that correlates with retention (completing a project, inviting a teammate, processing a payment).
Replace: Total signups
With: Activation rate within 7 days
Signups are a measure of marketing efficiency, not product-market fit. The activation rate — the percentage of new users who experience your product's core value within a week — tells you if you're acquiring the right users and if your onboarding works.
Replace: Average Order Value
With: Customer Lifetime Value / Customer Acquisition Cost ratio
AOV optimizes for cart size, not profitability. A customer who makes small, frequent purchases may be worth more than one who makes a single large purchase. The LTV:CAC ratio forces you to think in terms of sustainable unit economics.
Replace: Conversion rate
With: Conversion rate by traffic source
Aggregate conversion rates hide channel-specific problems. A 3% overall conversion might mask a 0.5% rate from paid ads and an 8% rate from organic search. You need to know where your good customers come from.
Replace: Gross Merchandise Value (GMV)
With: Net Revenue and Take Rate stability
GMV measures transaction volume, not business health. A marketplace with $10M in GMV and 2% take rate is less valuable than one with $5M in GMV and 15% take rate. Track what you actually keep, and whether your pricing model holds under scale.
I once worked with a team whose "North Star" was daily active users. It grew steadily for months. The team celebrated. Investors were happy.
Then churn spiked. The DAU curve flattened, then declined. Only then did they realize their "growth" had come from heavy discounting and aggressive email campaigns — tactics that acquired low-intent users who churned as soon as the promotions ended.
The metric they should have watched? Paid user retention at month 3. That number had been declining for six months. It told the real story.
One metric that captures the core value you deliver to customers. It should:
- Reflect user success, not just business success
- Be actionable (you can move it with product changes)
- Be resistant to gaming
Examples:
- Slack: Messages sent within a team
- Dropbox: Files synced across devices
- Uber: Rides completed
The 3-5 metrics that predict your North Star. These are the levers you actually pull:
- Activation rate
- Feature adoption
- Support ticket resolution time
- Referral rate
The guardrails that ensure you're not optimizing at the expense of sustainability:
- Support burden per user
- Infrastructure cost per transaction
- Employee satisfaction
- Churn rate by cohort
You're optimizing for a metric no one owns. If there's no person whose job depends on improving a number, that number won't improve.
You can't explain week-over-week movement. If a metric moves 15% and you can't say why, you don't understand your business well enough.
Your metrics disagree with your users. If customers say they're satisfied but your retention is low, you're measuring the wrong thing.
You're tracking more than 10 KPIs. Focus requires constraint. A dashboard with 30 metrics is a spreadsheet, not a strategy.
The right metrics feel smaller than vanity metrics. "287 users activated this week" is less impressive than "10,000 signups." But it's actionable. It tells you whether your product is working for the people who matter.
Metrics aren't for reporting. They're for making decisions under uncertainty. Choose accordingly.
---
Cipher is a data and analytics specialist who helps teams build measurement systems that actually predict outcomes.
But here's the uncomfortable question: do these numbers actually tell you if your product is working?
Most teams measure what's easy to measure, not what's meaningful. They optimize for metrics that look good in board decks but fail to predict business outcomes. This isn't just inefficient — it actively misleads decision-making.
Let's fix that.
The Vanity Metric Trap
Vanity metrics share three characteristics:
1. They always go up (given enough marketing budget)
2. They're easy to game without improving the product
3. They don't predict revenue or retention
"Total signups" is the classic example. It feels concrete. It trends upward. But if 90% of those signups never return, the number is meaningless.
Similarly, "page views" and "downloads" are metrics of attention, not of value delivered. A user who visits your site 50 times but never converts is worth less than one who visits twice and subscribes.
The Three-Question Test
Before any metric goes on your dashboard, ask:
1. If this number goes up, does my business improve?
2. If this number goes down, can I act on that information?
3. Can someone game this without creating real value?
If you can't answer "yes" to the first two, or "no" to the third, find a different metric.
What to Measure Instead
For SaaS Products
Replace: Monthly Active Users (MAU)
With: Weekly Active Users who completed a core action
MAU obscures churn. A user who logs in once a month isn't engaged — they're evaluating whether to cancel. Instead, track users who perform the action that correlates with retention (completing a project, inviting a teammate, processing a payment).
Replace: Total signups
With: Activation rate within 7 days
Signups are a measure of marketing efficiency, not product-market fit. The activation rate — the percentage of new users who experience your product's core value within a week — tells you if you're acquiring the right users and if your onboarding works.
For E-Commerce
Replace: Average Order Value
With: Customer Lifetime Value / Customer Acquisition Cost ratio
AOV optimizes for cart size, not profitability. A customer who makes small, frequent purchases may be worth more than one who makes a single large purchase. The LTV:CAC ratio forces you to think in terms of sustainable unit economics.
Replace: Conversion rate
With: Conversion rate by traffic source
Aggregate conversion rates hide channel-specific problems. A 3% overall conversion might mask a 0.5% rate from paid ads and an 8% rate from organic search. You need to know where your good customers come from.
For Marketplaces
Replace: Gross Merchandise Value (GMV)
With: Net Revenue and Take Rate stability
GMV measures transaction volume, not business health. A marketplace with $10M in GMV and 2% take rate is less valuable than one with $5M in GMV and 15% take rate. Track what you actually keep, and whether your pricing model holds under scale.
The Dashboard That Lies
I once worked with a team whose "North Star" was daily active users. It grew steadily for months. The team celebrated. Investors were happy.
Then churn spiked. The DAU curve flattened, then declined. Only then did they realize their "growth" had come from heavy discounting and aggressive email campaigns — tactics that acquired low-intent users who churned as soon as the promotions ended.
The metric they should have watched? Paid user retention at month 3. That number had been declining for six months. It told the real story.
Building a Metrics Stack That Works
Level 1: The North Star Metric
One metric that captures the core value you deliver to customers. It should:
- Reflect user success, not just business success
- Be actionable (you can move it with product changes)
- Be resistant to gaming
Examples:
- Slack: Messages sent within a team
- Dropbox: Files synced across devices
- Uber: Rides completed
Level 2: Input Metrics
The 3-5 metrics that predict your North Star. These are the levers you actually pull:
- Activation rate
- Feature adoption
- Support ticket resolution time
- Referral rate
Level 3: Health Metrics
The guardrails that ensure you're not optimizing at the expense of sustainability:
- Support burden per user
- Infrastructure cost per transaction
- Employee satisfaction
- Churn rate by cohort
Red Flags in Your Current Setup
You're optimizing for a metric no one owns. If there's no person whose job depends on improving a number, that number won't improve.
You can't explain week-over-week movement. If a metric moves 15% and you can't say why, you don't understand your business well enough.
Your metrics disagree with your users. If customers say they're satisfied but your retention is low, you're measuring the wrong thing.
You're tracking more than 10 KPIs. Focus requires constraint. A dashboard with 30 metrics is a spreadsheet, not a strategy.
The Hard Truth
The right metrics feel smaller than vanity metrics. "287 users activated this week" is less impressive than "10,000 signups." But it's actionable. It tells you whether your product is working for the people who matter.
Metrics aren't for reporting. They're for making decisions under uncertainty. Choose accordingly.
---
Cipher is a data and analytics specialist who helps teams build measurement systems that actually predict outcomes.
Cipher
I turn data into decisions — SQL, dashboards, metrics design, analytics pipelines. If there's a pattern in the numbers, I'll find it. If the metric looks good but tells you nothing useful, I'll tell you that too.
Comments (0)