CBSplit CMS Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Why winning a split test can lose money ======

===== The winner paradox =====

Split tests are designed to produce winners.

Higher conversion rate.
Better EPC.
Improved front-end revenue.

Traffic is shifted to the winner.
Budgets are increased.
Scaling begins.

And sometimes, profit declines.

CBSplit was built to explain this paradox.

===== Split tests optimize events, not outcomes =====

Most split tests measure:

  
  * Click-through rate
  * Conversion rate
  * Immediate revenue
  * Cost per acquisition

They rarely measure:

  
  * Refund-adjusted revenue
  * Rebill survival
  * Long-term churn
  * Processor risk

An event-level winner is not always an outcome-level winner.

===== Aggressive variants often win early =====

Variants that use:

  
  * Urgency pressure
  * Scarcity framing
  * Simplified billing disclosure
  * Emotional triggers

Often increase initial conversions.

They can also increase:

  
  * Refund likelihood
  * Customer regret
  * Early churn
  * Support friction

Winning the test may plant the seeds of loss.

===== Front-end revenue can mask backend erosion =====

A winning variant may:

  
  * Increase average order value
  * Boost upsell acceptance
  * Raise short-term EPC

If backend stability declines:

  
  * Refund rates rise
  * Rebills fail
  * Chargebacks increase

Net revenue drops over time.

CBSplit recalculates performance after backend resolution.

===== Short test windows distort long-term profitability =====

Split tests often run in short windows.

Short windows favor:

  
  * Immediate behavior
  * Impulse-driven buyers
  * Early revenue spikes

Long-term profitability depends on:

  
  * Customer alignment
  * Subscription durability
  * Refund containment

A short-term winner can become a long-term liability.

===== Blended metrics hide cohort differences =====

Winning variants may perform well overall while:

  
  * Failing in specific geographies
  * Producing high refunds in certain segments
  * Causing churn in particular cohorts

Blended results hide structural weaknesses.

CBSplit preserves segmentation to expose them.

===== Scaling magnifies small weaknesses =====

When a winner is scaled:

  
  * Traffic volume increases
  * Refund clusters grow
  * Processor scrutiny rises
  * Risk compounds

Minor structural flaws become major financial problems.

CBSplit evaluates scalability before declaring victory.

===== Profitability is net, not gross =====

True profit must account for:

  
  * Refund-adjusted revenue
  * Subscription retention
  * Payment stability
  * Acquisition cost

Winning a split test does not guarantee positive net margin.

CBSplit aligns test evaluation with net revenue truth.