Modern Approach to Testing Promotional Efficiency

A structured methodology for determining whether weekday or weekend promotions are more effective

1
Hypothesis
2
User Groups
3
Metrics
4
Implementation
5
Analysis
6
Iteration
7
Considerations

Hypothesis:

"Offering the same promotion on weekends (Group W) leads to higher net benefit—e.g., higher order frequency or higher average order value—compared to offering it on weekdays (Group D)."

What Are You Testing?

You want to see if sending promotions only on weekends outperforms sending promotions only on weekdays, or vice versa.

Why This Matters:

If weekends see more free time and group dining, you might expect higher basket sizes or redemption rates.
If weekdays see lunch/commute orders, that might also be an advantage.

Business Objective:

Maximize ROI of promotions (incremental revenue minus coupon costs).
Increase user engagement (repeated or more frequent orders).
Avoid "coupon fatigue" by placing promotions at the right times.

2.1. Identify the User Pool

Who is Eligible?

Typically, active users who have ordered at least once in the past 1–3 months.
Exclude users who already have special loyalty discounts or campaigns that might confound the results.

2.2. Randomly Assign Users to Groups

Group Description Promotion Time
Group D (Weekday) Receives promotional messages/coupons Monday–Thursday
Group W (Weekend) Receives promotional messages/coupons Friday–Sunday
Group C (Optional) No new promotion (baseline) N/A

2.3. Modern Tools & Methods for Group Splitting

Experimentation Platforms: Such as Optimizely, LaunchDarkly, Adobe Target, or in-house platforms that handle user-level randomization.
Randomization: Ensures each user has an equal probability of landing in each group.
Pre-Check for Baseline Similarities: Use historical data to ensure average order value (AOV), order frequency, or demographic splits are not significantly different between groups.

Choose metrics that capture coupon effectiveness and business impact:

Redemption Rate

Redemption Rate = (# of users who redeemed the coupon) ÷ (# of users who received the coupon) Measures how successful your offer was in driving customer action

Average Order Value (AOV)

Average order size (in $) during the experiment period.

Incremental Revenue

Additional revenue driven by the promotion (vs. baseline or historical values).

Order Frequency

The number of orders per user over the test window (e.g., average weekly orders).

Coupon Cost

If the promotion is a discount or free delivery, measure the cost to the business.

Net Profit or ROI

Net Profit or ROI = (Incremental Revenue - Coupon Cost) ÷ Coupon Cost Shows return on investment for every dollar spent on promotions

This is a direct business-centric metric.

Tip: Many modern data teams also track engagement metrics like "click-to-order time," "push open rates," or "cart additions," but the focus here is revenue/efficiency.

4.1. Set Up the Promotion Logistics

Promotion Content:

Same coupon type for both groups (e.g., 10% off or $5 off).
Send to Group D only on weekdays, Group W only on weekends.

Delivery Channel:

Decide whether to use push notifications, SMS, email, or in-app pop-ups.
Keep messaging consistent so only timing differs.

4.2. Instrumentation & Tracking

Event Tracking:

Capture "coupon viewed," "coupon redeemed," "order placed," "order value."

Attribution:

Ensure you can link each user's order back to the exact group assignment.

Modern Data Pipeline:

Tools like Segment, Amplitude, or Mixpanel can collect these events in real-time.

4.3. Duration of the Experiment

Minimum 2–4 Weeks:

Capture at least 2 weekend cycles to smooth out anomalies.
Let enough time pass so each user has multiple opportunities to redeem the offer.

Sample Size:

Use power analysis or built-in calculators in experimentation tools to estimate how many users need to be in each group.

5.1. Data Collection & Cleaning

At the end of the experiment:

Gather redemption events, total orders, total revenue per group.
Remove anomalies if needed (e.g., test accounts, fraudulent orders).

5.2. Compute Core Metrics

Redemption Rate

Redemption Rate = redeemed_users ÷ total_users_targeted

Average Order Value (AOV)

AOV = sum_of_all_orders_value ÷ count_of_orders

Order Frequency

Order Frequency = count_of_total_orders ÷ (total_users_in_group × # of weeks)

Incremental Revenue

Incremental Revenue = (Revenue in Test Group) - (Revenue in Control or Baseline)

Coupon Cost

E.g., if a coupon is $5 off and was redeemed 1,000 times, total cost = $5,000.

Net Profit / ROI

Net Profit = Incremental Revenue - Coupon Cost
ROI = Net Profit ÷ Coupon Cost

5.3. Statistical Significance Testing

Redemption Rate (categorical/binary outcome)

Use a z-test for proportions or a chi-square test.
Null Hypothesis (H₀): "Weekday redemption = Weekend redemption."

AOV (continuous outcome)

Use a t-test or non-parametric test depending on data distribution.
Null Hypothesis (H₀): "Mean AOV is the same for Weekday vs. Weekend."

Calculating Test Statistics

Z-Test for Proportions (Redemption Rate)

When comparing proportions (like redemption rates) between two groups:

z = (p₁ - p₂) / √[ p̂(1-p̂)(1/n₁ + 1/n₂) ]
p₁, p₂: Redemption rates in the weekday and weekend groups
n₁, n₂: Sample sizes of the two groups
p̂: Pooled proportion = (x₁ + x₂) / (n₁ + n₂) where x₁ and x₂ are the number of redemptions

If |z| > 1.96, the difference is statistically significant at a 95% confidence level (p < 0.05).

T-Test for Continuous Variables (AOV, Order Frequency)

For comparing means of continuous variables like Average Order Value:

t = (x̄₁ - x̄₂) / √[ s₁²/n₁ + s₂²/n₂ ]
x̄₁, x̄₂: Sample means of the two groups (e.g., mean AOV)
s₁², s₂²: Sample variances of the two groups
n₁, n₂: Sample sizes of the two groups

Compare the calculated t-value against a t-distribution with degrees of freedom calculated using the Welch-Satterthwaite equation.

5.4. Standard Error & Confidence Intervals

Understanding Standard Error

Standard error measures the precision of your sample statistics (like means or proportions) as estimates of the population parameters. Smaller standard errors indicate more precise estimates.

Standard Error for Proportions:
SE(p) = √[ p(1-p)/n ]
Standard Error for Means:
SE(x̄) = s/√n

Where p is the proportion, s is the sample standard deviation, and n is the sample size.

Calculating Confidence Intervals

Confidence intervals provide a range within which the true parameter is likely to fall:

95% CI = Estimate ± (1.96 × Standard Error)
For redemption rate: p ± 1.96 × SE(p)
For average order value: x̄ ± 1.96 × SE(x̄)

Always report confidence intervals alongside your point estimates to show the precision of your results.

Practical Considerations

Statistical significance (p < 0.05) merely suggests the observed difference isn't due to chance, but doesn't indicate business importance.
Consider practical significance: Is the difference large enough to justify changing your promotion strategy?
For small sample sizes, consider using more conservative significance levels (p < 0.01) to avoid false positives.
Always visualize your data distributions before applying tests to ensure their assumptions are met.

5.5. Interpret Results

If the Weekday group shows a significantly higher net revenue at acceptable cost, you might focus future promos on weekdays.
If the Weekend group outperforms, direct more marketing efforts to Friday–Sunday.
If no significant difference, consider other factors—maybe split by different user segments or test bigger changes in promotion value.

Example Interpretation

Results:

  • • Weekend group redemption rate: 28.5% ± 2.1% (95% CI)
  • • Weekday group redemption rate: 22.3% ± 1.9% (95% CI)
  • • Z-statistic: 3.42 (p-value: 0.0006)

Interpretation:

The weekend promotion had a significantly higher redemption rate (6.2 percentage points higher) with a very small p-value (p < 0.001), indicating this difference is highly unlikely to be due to chance. The non-overlapping confidence intervals further support this conclusion. Given these results, we would recommend shifting more promotional budget to weekends, assuming other metrics like AOV and ROI align with this finding.

Refinement

If weekends see better redemption but lower margin (maybe the coupon is too big), you could test a lower discount or try free delivery.
If weekdays yield decent revenue but not enough redemption, consider adjusting timing or coupon type.

Segmented Follow-Up Tests

You might discover certain user subgroups (e.g., high-frequency lunch crowd vs. occasional dinner crowd) respond differently to weekday vs. weekend promotions.
Perform a segment-based or multi-armed bandit approach where the system learns which segment responds best to which timing.

Long-Term Monitoring

Check if user behavior changes over time (e.g., does repeated exposure to weekend coupons reduce their novelty?).
Possibly rotate promotions to avoid user fatigue.

1. What Do We Mean by "Higher Net Revenue"?

In a promotional context, net revenue typically refers to the additional revenue a business earns after subtracting direct costs associated with the promotion. In the case of a discount or coupon, costs could include:

Discount Amount: e.g., if you give a $5 discount on a $20 order, your revenue is $15, but the "cost" of that coupon is $5.
Waived Delivery Fees: e.g., if free delivery normally costs $3 to the user, that $3 is now a cost the company absorbs.
Any Other Direct Promotional Costs: rebates, loyalty points, etc.

The Formula

Net Revenue = (Gross Revenue from Orders) - (Direct Promotional Costs)

Why Net Revenue Matters:

A promotion might drive a lot of orders (high redemption rate), but if the cost (discount) is too large, your profit margin shrinks. Net revenue is a bottom-line measure of how much you truly earned after promotional expenses.

2. Conflict: Higher Net Revenue vs. Higher Redemption Rate

2.1. Scenario

Group A: Has a high redemption rate (lots of users redeemed the coupon), but the total order size per redemption might be relatively small, or the discount might be large. Consequently, net revenue might be moderate or even low because each redemption brings limited profit.
Group B: Has a lower redemption rate, but users who do redeem spend a lot more on each order, or the discount is smaller, so the net revenue is higher overall.

2.2. Which Should You Prioritize?

Business Objectives:
Profit-Focused: Typically, maximizing net profit or net revenue is more important than simply having a high redemption rate. If you give massive discounts and lose margin, you might have a 50% redemption rate but net less money.
User Growth or Engagement: If your short-term goal is to acquire or re-engage dormant users, you might place higher value on redemption rate (willing to take a short-term profit hit to build a user base).
Broader KPI Set:
You might look at lifetime value (LTV) of newly acquired users. High redemption rate with short-term losses could be acceptable if these new or reactivated users become high-value over time.
Evaluate retention metrics: Do these redeemed orders lead to repeated orders later?
Conclusion:

Most often, net revenue (or net profit) has the higher priority for established businesses looking at the bottom line. However, if the strategy is "growth at all costs," a higher redemption rate might be prioritized. The key is to align the metric with the business goal.

Modern Approach to Testing Promotional Efficiency • Step-by-Step Methodology