CBSplit CMS Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Why time-based bias affects split testing ======

===== The hidden variable =====

Split tests are often presented as neutral comparisons.

Two variants.
Equal traffic.
Measured results.

What is rarely acknowledged is time.

Time changes behavior.
Time changes traffic.
Time changes revenue outcomes.

CBSplit was built to account for time-based distortion.

===== Early data favors speed =====

The first phase of a test captures:

  
  * Immediate conversions
  * Impulse-driven buyers
  * Early enthusiasm
  * Narrow audience segments

These signals dominate short evaluation windows.

Long-term behavior has not yet surfaced.

Time-based bias rewards immediacy.

===== Refunds and churn are delayed signals =====

Refunds often occur:

  
  * Days after purchase
  * After product usage
  * After billing clarification
  * After subscription renewal

Churn becomes visible only after rebill cycles.

Short tests exclude these outcomes entirely.

CBSplit extends evaluation to include delayed revenue truth.

===== Traffic evolves over time =====

Traffic during a test is not static.

Over days or weeks:

  
  * Audience pools expand
  * Ad costs fluctuate
  * Platform algorithms shift
  * Creative fatigue emerges

A variant tested under one traffic phase may behave differently under another.

Time changes input conditions.

===== Early statistical significance can mislead =====

A result may reach statistical significance quickly.

But early significance reflects:

  
  * Limited cohort behavior
  * Narrow time exposure
  * Short revenue horizon

As time passes, outcome patterns can reverse.

CBSplit evaluates performance across lifecycle windows.

===== Subscription funnels amplify time distortion =====

Subscription funnels depend on:

  
  * Rebill survival
  * Cohort aging
  * Long-term LTV
  * Refund stabilization

None of these stabilize instantly.

Short tests introduce systematic bias against durable offers.

CBSplit accounts for lifecycle timing.

===== Scaling amplifies time-based error =====

When early winners are scaled quickly:

  
  * Refund impact grows
  * Churn becomes visible
  * Processor risk increases
  * Net revenue shifts

Decisions made before time-based signals mature often require reversal.

CBSplit aligns decision speed with revenue maturity.

===== Time is not noise =====

In many dashboards, delayed data is treated as lag.

In reality:

  
  * Time reveals alignment
  * Time exposes fragility
  * Time confirms durability

Ignoring time-based bias creates distorted conclusions.

CBSplit treats time as a core variable.