The Long-Horizon Holdout Protocol

Overview

A measurement framework that challenges short-term growth wins by maintaining long-term control groups (holdouts) to verify if immediate lifts translate to sustainable value over 1-3 years.

Core principle: 30-40% of short-term "wins" show neutral or zero impact long-term.

The Process

┌─────────────────────────────────────────────────────────────────┐
│  T-0 (Launch)                                                   │
│  Split Traffic: 90% Treatment / 10% Long-term Holdout           │
├─────────────────────────────────────────────────────────────────┤
│  T+3 Weeks                                                      │
│  Initial Decision: Ship if short-term positive/neutral          │
│  Keep holdout running                                           │
├─────────────────────────────────────────────────────────────────┤
│  T+6 Months                                                     │
│  Automated Ping: Check retention and emerging GMV patterns      │
├─────────────────────────────────────────────────────────────────┤
│  T+12-18 Months                                                 │
│  Final Reckoning: Compare Cohort GMV                            │
│  If neutral/negative: deprecate feature or pivot strategy       │
└─────────────────────────────────────────────────────────────────┘

Key Principles

Common Mistakes

• Declaring victory after 2 weeks of significance
• Assuming neutral short-term = failure (might compound later)
• Not setting up infrastructure for long-term tracking

Source: Archie Abrams (Shopify VP Product & Growth) via Lenny's Podcast