Designing a Test for Checking Order Lift

A systematic approach to measure how coupons and promotions impact your Average Order Value

[Icon: Users] 1. Identify Your Target Population

Eligibility Criteria

All User Segment Options:

All Users

All users who have ordered at least once in the past 3 months.

Dormant Users

Dormant users who haven't ordered in the last 30 days.

High-Value Users

High-value users only (top 10% spenders).

Why is target population important?

Proper segmentation ensures your test measures lift in the most relevant population.

Sample Size Calculation

10%
5% 15% 25%
Expected Lift:

10%

Recommended Sample Size:

1,600 users

Why is this important?

Ensures enough users to detect a given lift with statistical significance.

Power Analysis Visualization:

5% Lift:
6,400 users
10% Lift:
1,600 users
15% Lift:
711 users
20% Lift:
400 users
25% Lift:
256 users
The smaller the lift, the larger the required sample size.

Key Points:

  • Clear target population definition is crucial.
  • Sample size depends on the minimum detectable effect.
  • Larger samples increase statistical power.

[Icon: UserCheck] 2. Form Your Control & Experiment Groups

Assignment Methods

Random Assignment (Typical A/B Test)

Randomly assign users to Control and Experiment groups.

Why: Balances user characteristics; works well with large samples.

Stratified Random Sampling

Divide users by baseline AOV range and assign randomly within each bucket.

Why: Ensures similar baseline metrics.

Matched Pairs

Pair similar users and randomly assign one to each group.

Why: Most precise for small user pools.

Key Point:

Both groups should have similar baseline AOV for valid results.

Distribution Visualization

Control Group
$50
AOV
Experiment Group
$55
AOV

AOV Distribution by Group:

Control Group
$20 $50 $80
Experiment Group
$20 $55 $80

Assignment Method Impact:

[Icon: AlertTriangle]
Random Assignment: May result in uneven baselines.
[Icon: Check]
Stratified Sampling: Yields more balanced groups.

[Icon: Activity] 3. Validate Similar Baseline AOV

Set Baseline Values

$50
$55

Pre-Test Analysis:

Calculate mean and standard deviation of AOV for both groups.

Control Statistics
Mean AOV: $50.00
Std Dev: $15.00
Experiment Statistics
Mean AOV: $55.00
Std Dev: $15.00

Calculate Mean & Std Dev:

Steps:

  1. Gather pre-test AOV data
  2. Compute average AOV
  3. Compute standard deviation
  4. Validate statistical significance

Statistical Significance

Baseline Difference:

$5.00
0 (Identical) $5 (Threshold) $10+

Test Result:

[Icon: Check] Not Significant

Validation Result:

[Icon: Check] Good to proceed

p-value is above threshold (0.05); no significant baseline difference.

Statistical Significance Test:

Perform a t-test (or alternative) on the baseline AOV data.

Goal: p-value > 0.05

Key Validation Points:

Why Validate?

Ensures both groups are comparable for valid testing.

If Validation Fails?

Re-run assignments or use stratification.

[Icon: Zap] 4. Run the Test & Measure Order Lift

Apply Different Treatments

Control Group

Standard Experience

No new coupon; standard experience.

[Icon: Clock] Track 2-4 weeks
$50 AOV

Experiment Group

NEW PROMO!

Receives the new promotional offer.

[Icon: Clock] Track 2-4 weeks
$55 AOV

Before & After Comparison:

Before:
Control: $50
Experiment: $50
After:
Control: $50
Experiment: $55

Track Orders During Test

  • •
    Run for 2-4 weeks to capture multiple orders.
  • •
    Avoid holidays that may skew results.
  • •
    Track all orders in both groups.

Calculate Order Lift

Lift Formula:

Lift = ((AOVexperiment - AOVcontrol) / AOVcontrol) × 100%
Control AOV
$50.00
VS
Experiment AOV
$55.00
Calculated Lift:
+10.00%

Evaluate Statistical Significance

Perform a t-test (or alternative) to confirm the difference in AOVs.

[Icon: Check]
Significant Lift
With a lift of 10.00%, the promotion shows a significant AOV increase.

Interpretation of Results:

Consider:

  • •
    Statistical significance: p-value < 0.05?
  • •
    Practical significance: Is the lift large enough?
  • •
    ROI: Does the increased AOV justify the cost?

The Math Behind Order Lift:

Mathematically,

Lift = ((AOVexperiment - AOVcontrol) / AOVcontrol) × 100%

For example, if Control AOV = $50 and Experiment AOV = $60, then:

Lift = (($60 - $50) / $50) × 100% = 20%

[Icon: Target] Test Summary & Key Insights

Target Population

All User Segments

(Dormant, high-value, etc.)

Sample Size

1,600 users

To detect a 10% lift

Methodology

Random, Stratified, or Matched

For balanced groups

Results

+10.00% Lift

Statistically significant

Key Takeaways

[Icon: Check]

Proper group selection is essential.

[Icon: Check]

Similar baselines validate accuracy.

[Icon: Check]

Statistical significance confirms non-random lift.

[Icon: Check]

Meaningful lift guides final decisions.