Your dashboard shows conversions climbing after you launched that new journey, and revenue is up. But one question decides whether the number means anything: did your messages cause that lift, or would those users have converted anyway?
Plenty of things move conversions that have nothing to do with you — a seasonal spike, a store feature, organic growth. There’s one way to know for sure: hold a slice of your users out of messaging entirely and watch what they do on their own. That’s exactly what a control group does.
This post covers how a control group works, how to read uplift, and how to set one up in Pushwoosh in a couple of clicks.
Why an A/B test can’t answer this
Faced with that question (how to measure messaging impact), most teams reach for A/B testing. It’s a classic tool, and a good one — it just isn’t built for this task.
An A/B test answers a narrower question: which message performs better. Push A versus Push B, urgency versus discount, push versus email. It’s the right tool for optimizing what you send, which is a different job from proving the sending was worth it.
That baseline is the thing you’re missing. Without it, you’re comparing your winners against your other winners, with no read on whether any of them beat silence.
| A/B test | Control group | |
|---|---|---|
| Question | Which message works best? | Does messaging work at all? |
| Who gets messaged | Everyone (all variants) | Everyone except a holdout |
| What you learn | The best-performing version | Your real, incremental impact |
People mix up A/B testing and control groups all the time. The problem is that confusion quietly burns budgets. While an A/B test tells you which message won, a control group tells you whether you should’ve hit send at all.
Opens, click rates, delivered rates, subscriber counts go up, everyone’s happy, and no one checks whether those users would have converted on their own.
Yes, a million subscribers and a 99% delivered rate prove your pipes work. But does that mean your messaging actually moved a single dollar? Conversions always go up: payday, seasonality, or just organic growth. If you can’t subtract that out, you’re taking credit for a result you didn’t earn.
Also, in an A/B test, you compare messaging options but never measure against silence. A control group does exactly that: hold a slice back, message everyone else, and the difference is what you actually drove in revenue, against a real baseline.
To sum up, A/B testing does make a message better, but a control group proves your team actually should exist.
What a control group actually measures
A control group (a holdout) splits your audience into 2:
- Treatment. Everyone who receives your marketing messages as usual: push notifications, email, in-app message.
- Control. A fixed slice held out from all marketing. They still get transactional messages like password resets and receipts, so nothing breaks for them. They just never receive a campaign.
Because the only difference between the 2 groups is your messaging, any gap in their behavior is your doing. That gap has a name: uplift.
In Pushwoosh, you can use the Global control group feature. You set a fixed percentage of users to hold out (1–15%), and the platform automatically compares the 2 groups on whatever conversion event you choose and provides you with analytics:
What this looks like in practice
Say you hold out 5% of your users and pick a conversion goal — the action that counts as success in your app. The conversion goal doesn’t necessarily have to be a purchase: a checkout success, a plan upgrade, a subscription started, a completed level, anything you treat as a win. Where you can, pick the one closest to revenue — that’s the number that carries weight when you’re defending a budget.
Over a 30-day window, the messaged 95% (treatment) run against the silent 5% (control), and you compare how each group converts:
| Group | Conversion rate |
|---|---|
| Treatment (messaged) | 8.2% |
| Control (held out) | 6.1% |
Uplift: +34%, marked Significant. (*Uplift is measured against the control baseline: (8.2 − 6.1) / 6.1 ≈ 34%.)
How to interpret it: Conversions driven by messaging: roughly 940 extra purchases in 30 days. At a $45 average order value, that’s about $42K in revenue your program can claim — the difference between “we sent a lot of pushes” and “we made the company $42K.”
That’s the happy path, and that’s how you prove to your CEO and CFO that your campaigns actually work! 💰
However, the number won’t always come back looking like this. Here’s how to read whatever it shows.
Reading the result: 3 outcomes and your next move
Once you have uplift, the number tells you what to do next.
| Result | What it means | Your move |
|---|---|---|
| Positive & significant (like in our example) | Your messaging creates real lift beyond the baseline | Scale the strongest flows, cut the weak. Take the number to the budget review — “a 34% lift over control” is a sentence no one waves off |
| Flat or not significant | Treatment and control convert about the same; those users were converting anyway | Rework the offer or tighten the segment before you cut. Usually it's a generic blast a segmented version would beat |
| Negative | Control converts better — the messaged group is worse off, usually from over-messaging | Cut frequency hard and re-measure. A control group is the only thing that catches over-messaging at all; without it, it looks fine on every open-rate chart you own |
Applying a Global control group is a must if you want to see clearly how your marketing communications land with your audience, and what happens next, compared to the users who are excluded from them.
This approach has proven its value with our customers in Fintech and Content Creation: adopting a Global control group gave them plenty of meaningful insights, action items, and adjustments they could bring straight into day-to-day sending operations.
The difference between the 2 groups — measured in industry-specific events, user actions and, conversely, signs of inactivity — works like a litmus test for your whole communication strategy. It’s especially useful when you feel you’ve run out of fresh ideas for your audience: the control group shows what your messaging is actually driving, and what users would have done anyway.
Set up your control group in Pushwoosh in a few clicks
-
Choose your holdout size
Set the share of users to hold out; 5% is the balanced default. New sign-ups join the group automatically.
-
Mark your campaigns as marketing
The holdout applies to marketing messages, so your transactional sends — receipts, password resets — still reach everyone.
-
Pick the events that define success
Add them as conversion goals in your journeys so the report knows what to measure. It doesn't have to be a purchase — pick whatever marks real value for your app: a completed purchase for e-commerce, an upgrade to paid for subscription, a completed level for gaming, or any other event you consider a conversion.
-
Read the report
The control group page shows your uplift and whether it's significant — no manual math on your side.
