A comprehensive framework for designing, analyzing, and optimizing promotion strategies
"Offering a specific promotion (e.g., 20% off or free delivery) will increase total orders and revenue by at least X% and improve repeat purchase rates within 30 days."
Set clear business goals and hypothesis
Select primary and secondary metrics
Create A/B test with proper randomization
Set up data pipeline and tracking
Calculate statistical significance and ROI
Deploy winning promotion or refine
Make promotion decisions based on statistical evidence rather than intuition
Identify which promotions generate positive ROI and which cost more than they're worth
Deliver the right promotions to the right customers at the right time
Establish a framework for ongoing testing and refinement
Definition: Percentage of users who place an order after seeing the promotion
Why it matters: Captures direct effectiveness (did the promotion lead to an immediate transaction?)
Definition: Mean spend per order in the promotion period
Why it matters: You want to make sure the promotion doesn't simply decrease margins by encouraging smaller orders
Definition: Additional revenue generated among users exposed to the promotion versus a control group
Why it matters: Shows net lift in revenue, factoring in the discount cost
For Test group (with promotion):
Net AOV (Test):
A promotion can boost gross AOV but yield a negative overall return if the discount eats into profit more than the additional revenue gained.
Shows monthly comparison between:
This chart shows why it's crucial to compare net AOV (after subtracting promotion costs) rather than just gross AOV. The test group appears to have higher AOV before accounting for promotion costs.
Month-over-month comparison:
Conversion rate is one of the most common metrics in food delivery promotion testing. It directly shows whether the promotion persuades more people to place an order.
Both groups should have equal baseline AOV (~$25) before randomization
No promotion (baseline)
Baseline AOV
$25.00
Gross = Net (no discount)
Receives 20% promotion
After Promotion
$22.40
$28 gross - $5.60 discount
Randomly split users into control and test groups to ensure there are no systematic differences between the groups.
Use power analysis to determine how many users you need to detect the expected effect size with confidence.
If different user segments may respond differently, consider stratified randomization.
Run the experiment long enough to account for user ordering cycles and business seasonality (typically 1-4 weeks).
Where:
Track who sees the promotion, clicks, browses
Store in Snowflake, BigQuery with experiment tags
Transform data, calculate metrics, visualize
Use LaunchDarkly, Optimizely, or custom systems to control who sees promotions
Store user interactions and orders in real-time (e.g., Snowflake, BigQuery)
Log: user ID, timestamp, promotion flag, order details, amount, discount, etc.
Use Airflow, dbt to extract experiment data, transform into consistent schema
Tableau, Looker, Metabase for daily metrics monitoring
H₀ (Null): The promotion has no effect
H₁ (Alternative): The promotion affects the metric
For each group (test vs. control), collect:
Determine probability of observing this difference by chance
If p < 0.05, reject H₀ and conclude the promotion had an effect
For Test Group (with promotion):
GOV (Gross Order Value): Amount the user pays before discounts
DA (Discount Amount): Cost of the promotion
NOV (Net Order Value): What company actually earns (NOV = GOV - DA)
For Control Group (no promotion):
Order Value (OV): Since no discount is applied, Gross = Net
For Test Group:
TotalNetRevenue_test = Σ(NetSpend_i) for all users i in test group
For Control Group:
TotalRevenue_control = Σ(OrderValue_i) for all users i in control group
For Test Group:
NetAOV_test = TotalNetRevenue_test / NumberOfOrders_test
For Control Group:
AOV_control = TotalRevenue_control / NumberOfOrders_control
ΔAOV = x̄_test - x̄_control = $22.4 - $25 = -$2.6
t = (x̄_test - x̄_control) / √(s²_test/n_test + s²_control/n_control)
t = (-2.6) / √((11²/2100) + (10²/2000))
t = -2.6 / √(0.058 + 0.05) = -2.6 / √0.108 = -2.6 / 0.329 = -7.9
p-value = 0.03 (well below 0.05 threshold)
Interpretation: The promotion increased gross AOV but decreased net AOV after accounting for discount costs. This difference is statistically significant (p < 0.05), meaning the promotion is actually hurting net revenue per order.
A p-value less than 0.05 means there's less than a 5% chance that the observed difference between test and control groups occurred by random chance.
Statistical significance doesn't always equal business relevance. Even tiny differences can be statistically significant with large sample sizes.
This knowledge graph visualizes the key components and their relationships in promotion effectiveness testing.
Your business goals directly determine which metrics are most important to track. For example, if customer retention is the goal, then repeat purchase rate becomes a primary metric.
The metrics you choose influence how you design your experiment, including sample size calculations and test duration.
The way you design your experiment determines the appropriate statistical methods to use for analysis.
The results of your statistical analysis (including both statistical and practical significance) guide your ultimate business decision about whether to deploy a promotion.
An effective promotion testing framework integrates all these components in a coherent, end-to-end process that aligns with business goals while maintaining statistical rigor.
Instead of one-size-fits-all promotions, use ML to deliver personalized offers:
Train ML models to predict which users have the highest propensity to respond to different promotion types.
Group users based on behavioral patterns and tailor promotions to each segment.
Factor in time of day, day of week, weather, and other context to optimize timing.
Instead of static A/B testing, use adaptive algorithms:
Continuously allocate more traffic to better-performing promotions in real-time, minimizing opportunity cost.
Balance exploration (trying different promotions) and exploitation (using the best-performing ones).
More efficient than traditional A/B testing, especially in high-traffic environments with multiple variants.
Looking beyond immediate conversion:
Measure how promotions affect the long-term value of customers, not just immediate purchases.
Track users who redeemed promotions vs. those who didn't over extended periods (30/60/90 days).
Use statistical techniques to analyze time until customer churn between promotion groups.
When user-level randomization is not feasible:
Assign different geographic regions to test and control conditions.
Statistical technique that accounts for pre-existing differences between regions and time-based trends.
Select test and control regions with similar historical performance to increase comparability.