What you'll learn
The "leaky bucket" trap — converting users who churn a month later
When A/B testing makes sense — and what to do when your traffic is too small
Why understanding the whole business beats any fixed test sequence or "best practice"
Guardrail metrics that stop a "winning" test from quietly hurting the business
The metrics that actually matter: trial starts, proceeds per user, ARPU & LTV
The app-to-web gray zone — Apple, Stripe, and web checkout without tanking conversion
Timestamps
- 00:00
- Intro
- 00:51
- Meet Luke — from product to "the paywall guy"
- 03:02
- Why the paywall decides: app or business?
- 06:52
- The leaky bucket — converting users who churn
- 08:11
- When A/B testing makes sense (and when it doesn't)
- 13:33
- What to test first — context over "best practices"
- 19:30
- Reading past tests: flaws, pending trials & baselines
- 22:39
- When benchmark reports mislead you
- 25:08
- "Even if it's a win, what did we lose?" — guardrail metrics
- 28:44
- The metrics that matter: trial starts, RPU & LTV
- 31:01
- The app-to-web gray zone: Stripe, Apple & rejections
- 33:39
- Moving users to web checkout without losing conversion
- 38:25
- Prioritizing tests & where to steal ideas
- 45:22
- What makes a good paywall (value before the paywall)
- 50:04
- Personalization & segmentation: the dog-training app example
- 52:21
- Failures, humility & a healthy testing culture
- 57:30
- Where AI actually helps (and where it doesn't)
- 1:04:03
- Growing expertise when there's no playbook
- 1:06:19
- Wrap-up