Table of Contents
Why CBSplit audits split test outcomes instead
The difference between running and auditing
Running a split test is mechanical.
Create variants. Split traffic. Measure conversions. Declare a winner.
Auditing a split test is different.
It asks whether the declared winner actually improved the business.
CBSplit was built to audit outcomes, not just events.
Event wins are not outcome wins
Most split tests optimize:
* Click-through rate * Conversion rate * Front-end revenue * EPC
These are event-level metrics.
Outcome-level metrics include:
* Refund-adjusted revenue * Rebill survival * Subscription churn * Net lifetime value * Processor risk exposure
CBSplit audits whether event-level wins translate into outcome-level gains.
Auditing extends beyond the test window
Traditional tests end when:
* Statistical significance is reached * A performance threshold is crossed * A decision deadline arrives
CBSplit continues evaluating:
* Refund timing * Rebill cycles * Cohort durability * Delayed financial reversals
Outcome truth often emerges after the test officially ends.
Auditing reveals hidden reversals
A declared winner may:
* Increase initial conversions * Raise early ROAS * Improve surface-level metrics
An audit may reveal:
* Higher refund rates * Lower retention * Increased churn * Reduced net profit
Without auditing, these reversals remain invisible.
Auditing connects traffic source to lifecycle impact
Split tests rarely account for:
* Traffic quality differences * Segment-specific refund clusters * Geographic variance * Retry-dependent approvals
CBSplit audits results by segment, not just aggregate.
This prevents weak segments from hiding behind blended averages.
Auditing protects scaling decisions
Scaling decisions based on incomplete tests can:
* Amplify refund exposure * Increase processor scrutiny * Erode long-term margin * Create unstable revenue patterns
Auditing ensures that winners remain profitable under scale.
Auditing aligns experiments with business reality
A split test asks:
* Which variant performed better?
An audit asks:
* Did profitability improve? * Did customer quality improve? * Did lifecycle stability improve? * Did risk decrease?
CBSplit focuses on business significance, not just statistical significance.
Infrastructure over automation
CBSplit does not compete with A/B tools.
It complements them by:
* Validating conclusions * Recalculating net revenue * Extending evaluation timelines * Surfacing lifecycle risk
It ensures that testing decisions reflect financial reality.