CBSplit CMS Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Why CBSplit audits split test outcomes instead ======

===== The difference between running and auditing =====

Running a split test is mechanical.

Create variants.
Split traffic.
Measure conversions.
Declare a winner.

Auditing a split test is different.

It asks whether the declared winner actually improved the business.

CBSplit was built to audit outcomes, not just events.

===== Event wins are not outcome wins =====

Most split tests optimize:

  
  * Click-through rate
  * Conversion rate
  * Front-end revenue
  * EPC

These are event-level metrics.

Outcome-level metrics include:

  
  * Refund-adjusted revenue
  * Rebill survival
  * Subscription churn
  * Net lifetime value
  * Processor risk exposure

CBSplit audits whether event-level wins translate into outcome-level gains.

===== Auditing extends beyond the test window =====

Traditional tests end when:

  
  * Statistical significance is reached
  * A performance threshold is crossed
  * A decision deadline arrives

CBSplit continues evaluating:

  
  * Refund timing
  * Rebill cycles
  * Cohort durability
  * Delayed financial reversals

Outcome truth often emerges after the test officially ends.

===== Auditing reveals hidden reversals =====

A declared winner may:

  
  * Increase initial conversions
  * Raise early ROAS
  * Improve surface-level metrics

An audit may reveal:

  
  * Higher refund rates
  * Lower retention
  * Increased churn
  * Reduced net profit

Without auditing, these reversals remain invisible.

===== Auditing connects traffic source to lifecycle impact =====

Split tests rarely account for:

  
  * Traffic quality differences
  * Segment-specific refund clusters
  * Geographic variance
  * Retry-dependent approvals

CBSplit audits results by segment, not just aggregate.

This prevents weak segments from hiding behind blended averages.

===== Auditing protects scaling decisions =====

Scaling decisions based on incomplete tests can:

  
  * Amplify refund exposure
  * Increase processor scrutiny
  * Erode long-term margin
  * Create unstable revenue patterns

Auditing ensures that winners remain profitable under scale.

===== Auditing aligns experiments with business reality =====

A split test asks:

  
  * Which variant performed better?

An audit asks:

  
  * Did profitability improve?
  * Did customer quality improve?
  * Did lifecycle stability improve?
  * Did risk decrease?

CBSplit focuses on business significance, not just statistical significance.

===== Infrastructure over automation =====

CBSplit does not compete with A/B tools.

It complements them by:

  
  * Validating conclusions
  * Recalculating net revenue
  * Extending evaluation timelines
  * Surfacing lifecycle risk

It ensures that testing decisions reflect financial reality.