CBSplit CMS Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Why classic A/B testing lies in subscription funnels ======

===== The experiment illusion =====

Classic A/B testing assumes clarity.

Two variants.
One metric.
A winner.

In subscription funnels, this model breaks.

CBSplit was built because subscription revenue does not behave like single-purchase revenue.

===== Classic A/B testing measures the wrong moment =====

Most A/B tests declare winners based on:

  
  * Initial conversion rate
  * Front-end revenue
  * EPC
  * Cost per acquisition

Subscription funnels generate revenue over time.

The true outcome depends on:

  
  * First rebill success
  * Second rebill retention
  * Refund timing
  * Churn velocity

Classic A/B testing stops before these signals mature.

===== Early winners often lose later =====

Variant A may:

  
  * Convert more aggressively
  * Generate higher front-end revenue
  * Show stronger short-term metrics

Variant B may:

  
  * Convert slightly less
  * Produce stronger retention
  * Deliver higher lifetime value

Classic A/B testing crowns Variant A.

Rebills often crown Variant B.

CBSplit waits for lifecycle truth.

===== Subscription funnels amplify small differences =====

Minor differences in:

  
  * Messaging tone
  * Billing clarity
  * Upsell sequencing
  * Expectation framing

Can create large differences in:

  
  * Refund rates
  * Churn timing
  * Rebill survival

Classic A/B testing rarely runs long enough to capture compounding effects.

CBSplit tracks cohort durability.

===== Statistical significance ignores lifecycle complexity =====

Traditional A/B tests aim for statistical significance.

They assume:

  
  * Stable conditions
  * Immediate outcomes
  * Linear conversion paths

Subscription funnels are:

  
  * Non-linear
  * Delayed
  * Affected by retries and rebills
  * Influenced by fulfillment quality

Significance at checkout does not equal significance over time.

CBSplit evaluates outcome stability, not just event frequency.

===== Short test windows create structural bias =====

Short testing windows favor:

  
  * High-pressure copy
  * Urgency-driven conversions
  * Impulse behavior
  * Aggressive billing framing

These tactics often increase early revenue but weaken retention.

Classic A/B testing rewards speed.

CBSplit rewards durability.

===== Refund and churn distort true performance =====

Refunds and churn:

  
  * Reduce net revenue
  * Shift LTV curves
  * Alter ROI thresholds
  * Impact processor health

If these are excluded from evaluation, test results are incomplete.

CBSplit integrates refund-adjusted and rebill-adjusted outcomes.

===== Subscription truth is delayed by design =====

Subscription revenue unfolds over cycles.

Real performance becomes visible only after:

  
  * Multiple billing attempts
  * Cohort aging
  * Refund windows closing

Classic A/B testing assumes truth is immediate.

CBSplit assumes truth is delayed.