Experimentation & A/B Testing Skill

When to Use

• Validating new subject lines or creative.
• Testing segmentation hypotheses (persona vs behavior).
• Optimizing cadence, timing, or automation triggers.

Framework

1. Hypothesis – define expected uplift + rationale.
2. Metric Selection – primary (open/click/conv) + guardrails (unsubs, spam).
3. Sample Sizing – ensure stat significance (min 500 recipients per variant or use power calculator).
4. Execution – randomize, keep variants isolated, limit simultaneous tests.
5. Analysis – use z-test or Bayesian uplift; document learnings.

Templates

• Experiment brief (hypothesis, segments, KPI, risk guardrails).
• Variant table (control vs test inputs, creative asset links, owner).
• Calculator sheet for minimum detectable effect + sample size.
• Post-test debrief doc capturing learnings + rollout plan.