CBSplit CMS Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Why statistical significance is not business significance ======

===== The significance illusion =====

In classic experimentation, statistical significance is treated as the final authority.

If a result is significant, it is declared valid.
If it is not significant, it is ignored.

In revenue systems, this logic is incomplete.

CBSplit was built because statistical clarity does not always equal business impact.

===== Statistical significance measures probability, not profit =====

Statistical significance answers:

  
  * Is the difference likely real?
  * Is the variation unlikely due to randomness?

It does not answer:

  
  * Does this increase net revenue?
  * Does this reduce refunds?
  * Does this improve subscription survival?
  * Does this protect processor health?

A statistically valid result can still lose money.

===== Small gains can be statistically real and financially irrelevant =====

A variant may show:

  
  * 1% higher conversion
  * Slightly better click-through
  * Marginal EPC improvement

With enough traffic, these differences become statistically significant.

But after:

  
  * Refund adjustment
  * Rebill evaluation
  * Acquisition cost reconciliation

The financial impact may disappear.

CBSplit evaluates net outcome, not just event difference.

===== Large business impact can appear statistically weak =====

Some improvements affect:

  
  * Churn reduction
  * Refund containment
  * Rebill stability
  * Long-term LTV

These effects:

  
  * Emerge slowly
  * Require longer observation
  * Produce delayed financial gains

They may not reach statistical significance in short test windows.

Yet their business impact can be substantial.

===== Revenue systems are non-linear =====

Statistical models assume:

  
  * Stable environments
  * Isolated variables
  * Clean measurement windows

Revenue systems involve:

  
  * Traffic shifts
  * Payment retries
  * Refund timing
  * Subscription churn
  * External platform changes

Significance calculated in isolation may ignore systemic effects.

CBSplit evaluates performance within operational context.

===== Statistical wins can increase risk =====

A statistically significant improvement may also:

  
  * Increase refund rates
  * Raise chargeback exposure
  * Stress payment systems
  * Attract low-quality traffic

Statistical models rarely account for downstream risk.

Business significance must include risk exposure.

CBSplit integrates risk-aware metrics.

===== Time horizon changes business meaning =====

Statistical tests often operate within:

  
  * Short time frames
  * Limited cohorts
  * Controlled conditions

Business outcomes unfold across:

  
  * Multiple billing cycles
  * Refund windows
  * Traffic scaling phases

CBSplit extends evaluation beyond short-term statistical windows.

===== Decision quality depends on economic impact =====

Business significance requires:

  
  * Net revenue improvement
  * Refund-adjusted profitability
  * Rebill durability
  * Cohort-based LTV growth
  * Processor safety

Statistical confidence without economic improvement is incomplete.