A/B Testing

Perform A/B tests on your audience to figure out which push configuration works better for you

A/B test helps you decide which push configurations work better for your audience. Create and send different versions of a push message to understand which is more relevant and leads to more conversions. Specify different content, actions performed after push is opened, segmentation or scheduling rules, and track the results!

Starting A/B test

To perform an A/B test, go to your application and click Start A/B Test button in the Send Push form:

You can create up to 4 versions of the push notification, and specify the different content and necessary parameters for each of them in corresponding fields — just like separate Send Push forms gathered on your screen:

Message variants within A/B test can differ by segmentation patterns and scheduling rules, and can perform different actions after the push notification is opened by a user. For each variant you can type the text or select a preset from those created previously. Each variant can be assigned to a campaign to track results comprehensively.

In case the campaign associated with the preset differs from the campaign set manually for an A/B test variant, the latter will prevail. That means the A/B test variant will be assigned to the campaign you selected from the drop-down in the top right-hand corner of the Send Push form.

To learn more about Campaigns, please refer to our guide.

For example, let’s run an A/B test of push messages containing a Deep Link or a Rich Media page to decide which Custom Action parameter works better for conversions. Let’s say we have two presets: “DeepLink Promo” preset assigned to “Cart Recovery”campaign and “RichMedia Promo” preset assigned to “Return Users” campaign. Once you select a preset for an A variant, the campaign field in the top right-hand corner is filled automatically with the campaign assigned to that preset.

Different variants within A/B test can be assigned to different campaigns. In case you select a campaign before starting A/B test, all the newly created variants will be assigned to that campaign unless you change it for a variant manually.

If you select another campaign from the drop-down manually, it will prevail over the preset’s campaign. In the example given, the A variant is assigned to the “Return Users” campaign instead of the “Cart Recovery” set for the “DeepLink Promo” preset.

Please note A/B test variants are considered as separate message configurations, so that to assign each variant to a campaign different from the preset’s one you should select a campaign for each variant separately.

Tracking results

After an A/B test is sent, you can view statistics for each version of the message in the Message History in Control Panel:

Compare the stats: a variant with a higher open rate is definitely more attractive to your subscribers.

You can also view the advanced details of every particular push. Just click the Details button.

The test results are also represented as a bar chart like on the screenshot below.

That's it! Perform A/B tests and find out what is relevant to your audience, and what is not.