Incrementality, or incremental lift, is a key metric for understanding the value of your retargeting campaigns in your overall marketing strategy. Our team has written much about the basics of incrementality and why it’s the best metric of evaluating campaign success. Below, we tackle how to set up a meaningful incremental lift test for your retargeting campaigns.

Why you should be testing incrementality

incrementality test ITT

To start, let’s define some terms. First: incrementality. The incrementality of a retargeting campaign measures the additive lift that the spend invested in re-engaging users contributed to your overall goal. An incrementality test, or incremental lift test, does this by creating a holdout or control group. This group of users is randomly selected and does not see ads the way the test group of users do.

In this way, an incrementality test takes the form of an A/B test that compares the impact of one activity versus another. However, an A/B test does not include a holdout group. So, while an A/B test will evaluate the relative performance of showing an ad, for example, with a CTA button in a certain color against the performance of an ad with a CTA button with no color; the basic format of incrementality testing measures the impact of showing an ad to a group of users against the impact of not showing an ad to a control group of users.

The outcome of incremental lift testing is a deeper understanding of the value of your retargeting campaigns. This can help you answer the question of whether your paid campaigns are generating ROI that wouldn’t  have happened organically. It can also help to prove the value of investing in retargeting campaigns to yourself, your team and stakeholders in your organization.

The basics of incrementality testing 

What your incrementality test measures will be based on your app’s goals. For example, if you’re growing a game app, you likely have revenue-related campaign goals such as a Day 30 ROAS goal and an expected cost per action (eCPA). If your goals are revenue related then your incrementality test should evaluate the incremental lift of revenue and ad spend as a result of your retargeting campaigns.

On the other hand, if you are a non-gaming app, your goals might be related to conversions. If this is the case then your incrementality test could measure the incremental lift of your conversion rate (CR), lift in unique conversions or the incremental revenue generated per a desired action, like a subscription sign up.

Finally, your retargeting partner should be able to measure incremental lift on both a cohort level and within a specific window of time. This helps you understand how your retargeting campaigns are performing within a target group of users and also within a specific period of time. Once you have your testing set up, it’s important run tests long enough to collect a sufficient amount of data. For example, at least 4 weeks.

The types of incrementality tests you can do

There are 3 basic methodologies for undertaking an incrementality test.

Highest level: Intent-to-treat (ITT)

ITT lift analysis is the most basic form of incremental testing. A treatment group receives an ad while a control group does not. The differing response of each group provides the lift analysis.

The reason why this type of analysis is the highest level is because it deals with attribution-level data rather than bid request or impression data. So, on a high level, you can see that users in the control group did not see an ad, while users in the test group did. But, you can’t see impressions or any user-level data that has to do with users that saw an ad, converted or did not convert.

Since ITT operates on a higher, overall audience level, it doesn’t take into account that not all users in the test group will be shown an ad. All users in the test group can be shown an ad but the reality is they won’t all be. The result is lift analysis data that is less concrete, more broad and “noisy”. Noisy data is data that includes meaningless information that comes from the unexposed population of users in the test group.

Second level: Ghost bids

The second level of incrementality testing goes a stage lower to the bid request level. User-level data is available, therefore, users can be segmented within the target group. This gives the ability to separate out the users in the test group that do not see an impression or receive a bid request. This solves the noisy data that can result from the unexposed users in the test group mentioned in the ITT methodology above. In contrast, ITT does not have user-level data. Therefore, the ability to segment out specific users from the test group that do not see an ad.

The ghost bid methodology places bids on users in the test group while also tagging them into two different groups: exposed (seen at least one impression) and unexposed (no auctions for this user were won so no impressions were rendered).

To further control for noisy data, it also places invisible bids on users in the control group. These “ghost bids” create two groups of users in the control group; users who ‘would have been exposed’ and ‘would not have been exposed’. Incremental lift is then measured by comparing the results of the exposed users in the test group and the users in the control group that ‘would have been exposed’. This creates an apples-to-apples comparison by only looking at the data of the users who fit in the segment and also who received a bid request or potentially could have received a bid request.

Third level: PSA ads

The third level goes a step further by serving real ads to both the test and control groups and measuring the outcome in results. This is for situations in which you don’t want to settle for evaluating users who saw a bid or had the potential to see a bid. This methodology serves a random selection of the test group with a brand-related ad and a public service announcement (PSA) to a random selection of the users in the control group. Example PSAs include a don’t drink and drive ad or a philanthropic call to action.

What this does is create an apples-to-apples testing environment by serving actual ads to both groups. This has the effect of eliminating the noisy data coming from the unexposed users in the test group.

There are downsides to this methodology however. Serving PSAs comes at a cost for advertisers, since they’re spending to serve non-branded ads. Arguably, the content of PSAs can also create biased results. For example, a user might react to a blood donation ad but not an ad teasing a mobile game. In other words, PSA testing makes the assumption that the control group’s behavior when shown a PSA is sufficiently comparable to that of the test group’s when served a branded ad, which might not be the case.

Why Ghost bids are the most cost-effective and efficient option for incrementality testing

incrementality test ghost bid

As we discussed above, the highest level of incremental testing — ITT —  comes at no additional cost to the advertiser. However, it does include the issue of noisy data.

In our opinion, the second-best option is the third level of testing: PSAs. This methodology creates a clean apples-to-apples comparison by serving real ads to both groups. However, it requires the advertiser to spend money serving ads that have no relevance to their brand.

Finally, the best option in our opinion is the second-level: ghost bids. Why? Because it creates an apples-to-apples comparison at no additional cost to advertisers. Instead of spending extra to serve a PSA, ghost bids maximize the use of programmatic technology by placing bids on users in the real time bidding (RTB) exchanges from both groups.

Takeaways on Incrementality Testing

The incrementality of a retargeting campaign measures the additive lift that your retargeting spend contributed to marketers’ overall goals. An incremental lift test does this by creating a holdout or control group. There are 3 basic methodologies for undertaking an incremental lift test.

  • ITT: A test group receives an ad while a control group does not. The differing response of each group provides the lift analysis. This type of analysis is the highest level because it includes attribution-level data rather than user-level data or bid request and impression data. Since this high level does not separate out users who do not see ads in the test group, the data is noisy and too broad to be meaningful.
  • PSAs: This methodology solves the noisy data issue by serving real ads to both the test and control groups. The ads served to the control group are public service announcements. While this type does create an apples-to-apples testing environment, it requires advertisers to pay to serve non-brand related ads.
  • Ghost bids: This methodology is the most cost effective and efficient of the three. It solves the noisy data issue by segmenting out users in the test group that are not served an ad. It then also places invisible ghost bids on users in the control group. This provides an apples-to-apples testing environment using programmatic technology without spending additional money on serving non-brand related ads.

Prove the value of your retargeting campaigns with incrementality testing

Looking for a deeper understanding of the value of your retargeting campaigns? An incremental lift analysis can be a great way to improve re-engagement and retention efforts by diving deeper into the elements of your campaigns that are driving performance. YouAppi can help you set up retargeting campaigns and incremental testing. Send us a message here!