Table of Contents
Why winning a split test can lose money
The winner paradox
Split tests are designed to produce winners.
Higher conversion rate. Better EPC. Improved front-end revenue.
Traffic is shifted to the winner. Budgets are increased. Scaling begins.
And sometimes, profit declines.
CBSplit was built to explain this paradox.
Split tests optimize events, not outcomes
Most split tests measure:
* Click-through rate * Conversion rate * Immediate revenue * Cost per acquisition
They rarely measure:
* Refund-adjusted revenue * Rebill survival * Long-term churn * Processor risk
An event-level winner is not always an outcome-level winner.
Aggressive variants often win early
Variants that use:
* Urgency pressure * Scarcity framing * Simplified billing disclosure * Emotional triggers
Often increase initial conversions.
They can also increase:
* Refund likelihood * Customer regret * Early churn * Support friction
Winning the test may plant the seeds of loss.
Front-end revenue can mask backend erosion
A winning variant may:
* Increase average order value * Boost upsell acceptance * Raise short-term EPC
If backend stability declines:
* Refund rates rise * Rebills fail * Chargebacks increase
Net revenue drops over time.
CBSplit recalculates performance after backend resolution.
Short test windows distort long-term profitability
Split tests often run in short windows.
Short windows favor:
* Immediate behavior * Impulse-driven buyers * Early revenue spikes
Long-term profitability depends on:
* Customer alignment * Subscription durability * Refund containment
A short-term winner can become a long-term liability.
Blended metrics hide cohort differences
Winning variants may perform well overall while:
* Failing in specific geographies * Producing high refunds in certain segments * Causing churn in particular cohorts
Blended results hide structural weaknesses.
CBSplit preserves segmentation to expose them.
Scaling magnifies small weaknesses
When a winner is scaled:
* Traffic volume increases * Refund clusters grow * Processor scrutiny rises * Risk compounds
Minor structural flaws become major financial problems.
CBSplit evaluates scalability before declaring victory.
Profitability is net, not gross
True profit must account for:
* Refund-adjusted revenue * Subscription retention * Payment stability * Acquisition cost
Winning a split test does not guarantee positive net margin.
CBSplit aligns test evaluation with net revenue truth.
