Proving the value of your paid growth campaigns continues to be one of the main challenges that app marketers face today. Why? Because there is no standardized industry calculation for measuring what drives engagement. As a result, growth marketers continue to face challenges answering the simple question: “are my campaigns generating return on investment that wouldn’t have happened organically?”
This is where incrementality testing comes in: the best measure for ad success. Evaluating incremental lift measures advertising campaigns individually for their effectiveness.
What is incrementality and how can you use it to determine the value of your retargeting campaigns? We answer all your questions below including how to execute a meaningful incrementality test for your retargeting campaigns.
1. What is incrementality?
Incrementality in marketing measures the incremental lift that advertising spend contributes to the overall conversion rate. An explanation of ROAS provides a useful comparison to understand incrementality. While ROAS calculates the overall effectiveness of advertising campaigns by dividing total revenue over the total cost of advertising campaigns; incremental lift analysis calculates the percentage of total conversions that resulted directly from an advertising campaign. You can measure lift in engagement, in-app spend, or conversion frequency.
2. Why is incrementality testing important?
Budget Optimization
Understanding the specifics of how your ad spend affects your campaign revenue is critical to allocating your ad spend effectively. In the same way that “correlation does not imply causation,” attributed conversions should not imply that your ad spend is resulting in actions that wouldn’t have happened organically. In short, marketers can use incrementality testing to determine the relationship between their revenue and their advertising spend.
Partner Evaluation
Along with budget optimization, incrementality enables marketers to assess their growth partners by measuring whether they’re adding value or simply generating events that would have happened organically.
Sets the Right Incentives
Finally, incremental lift analysis limits the creation of bad incentives. If you’re hyper-focused on certain goals and not on measuring the incrementality of your campaigns, you might over emphasize targeting users at a specific point in your sales funnel. For example, in the case of CPA-based campaigns, marketers target users who are most likely to undertake an action farther down in the sales funnel. That being said, an unintended consequence of this focus is that it doesn't leverage users outside of this stage of the funnel to the fullest.
3. How do I do an incrementality test?
The Importance of a Holdout Group
Incrementality testing uses the principle of RCT (randomized control trials) of comparing a treatment group (users that can be exposed to the ad campaign) with a control group (users that aren’t exposed to an ad campaign) to calculate the relative lift. Isolating a control group — or holdout group — of users that you do not serve ads to is key to determining what sales revenue would have been without an ad campaign. Comparing this data to a treatment group then allows you to specifically measure how effective an individual advertising campaign was.
A basic incrementality test starts with the random selection of a test group and a control group. Generally, the control group includes 10% of users and the test group, 90% of users. The test group receives an ad and the control group does not. The difference in the conversion rate of both populations is then measured for incremental conversions.
Different Incremental Testing Methodologies
Each type of lift analysis includes the isolation of a control (or holdout) group. But, they differ in the types of ads the treatment group sees and the evaluation of campaign results. Below, we cover the benefits and limitations of each type of incrementality marketing test.
Intent-to-treat (ITT)
This is the most basic methodology and essentially follows the main principles of lift analysis. A treatment group receives an ad while a control group does not. The differing response of each group provides the lift analysis. This method compares the behavior of all users in both groups. This includes both the exposed and unexposed users in the test group and the users in the control group. The key to keep in mind here is that while all users in the treatment group can be shown an ad, they won’t all be. This results in “noisy” data, or meaningless information that comes from the unexposed population of the treatment group.
The benefits & disadvantages of ITT
Most advertisers use this approach since it’s simple and low cost. However, this method can result in bias since the analysis does not include all users in the treatment group. This is further amplified in situations where campaign parameters result in low availability of the target audience or there’s a low win rate on programmatic auctions.
Public Service Announcements (PSAs)
As a solution to the noisy data issue mentioned for ITT testing, PSA testing serves real ads to both groups. That said, the ads differ. A random selection of the test group receives a brand-related ad while a random selection of the control group receives a public service announcement. This might include a don’t drink and drive ad or a philanthropic call to action.
The benefits & disadvantages of PSAs
Serving real ads to both groups eliminates unexposed users in the treatment group, thereby canceling out noisy data. PSAs also drive value by raising social awareness around important public service issues.
However, this incrementality testing methodology is not cost effective since marketers pay to serve non-branded ads to users. It also doesn’t provide an “apples-to-apples” comparison to allow for an unbiased evaluation of incrementality in marketing. A user might react to a blood donation ad but not an ad teasing a mobile game. In other words, PSA testing makes the assumption that the control group’s behavior when shown a PSA is completely comparable to that of the test group’s when served a branded ad.
Ghost Ads
Where ITT falls short because of noisy data and PSA testing falls short because it doesn’t provide an apples-to-apples comparison, ghost ads solve both challenges. Ghost ads provide a low noise and low selection bias environment. Marketers can also use ghost ads at no additional cost.
Ghost ads monitor a control group and flag when a brand’s ad would have been served to a user in that group. Control group users receive an ad from another advertiser on a platform, which removes the cost for clicks and impressions. The control group behavior is then marked with a “ghost impression”, giving information on which control group users would have been exposed to the ad.
The ghost ad itself is invisible to the user. This maintains a quality user experience. They simply see the ad that wins the auction. However, the ghost ad and its bid data are visible to advertising partners. They can see how users who should have seen an ad behave even though they didn’t actually see it.
The benefits & disadvantages of ghost ads
Again, ghost ads achieve the PSA-based A/B testing methodology, but they do it more precisely and affordably. Ghost ads provide an apples-to-apples comparison of users exposed to ads versus users who would have been exposed to ads.
Another benefit of ghost ad testing is that it can always be “on”. In other words, marketers can run a ghost ad campaign concurrently to a live campaign. This provides consistent lift analysis information.
While ghost ads mitigate issues of noisy data and create an apples-to-oranges comparison, they don’t work well for retargeting. Ghost ads require a second interested party for the user in question. This works in the UA space because multiple advertisers find the same users relevant. On the other hand, in retargeting, advertisers try to re-engage a very specific, narrow set of user segments. Therefore, the ghost ads methodology is not effective when it comes to testing the incrementality of retargeting campaigns.
Ghost Bids
Last but not least - ghost bids: the most precise and cost-effective incrementality test method for retargeting campaigns!
Similar to the methodology described with ghost ads, ghost bids show invisible ads to users in both the control and treatment groups to eliminate noisy data. Set-up also includes the removal of all users not found on the real time bidding (RTB) exchanges from both groups. This also significantly reduces noise.
This methodology includes tracking users found on the RTB ad exchanges in the treatment segment for revenue and conversions. These users receive bids. This creates two groups of users in the treatment group: exposed (seen at least one impression) and unexposed (no auctions for this user were won so no impressions were rendered).
This methodology places ‘Ghost bids’ — or invisible bids — on the users in the control group. Two groups are also created in the control group: users who ‘would have been exposed’ and ‘would not have been exposed’. Conversions from the exposed users in the treatment group are then compared to the ‘would have been exposed’ users in the control group to measure for incremental lift.
The benefits of ghost bids
Eliminating users outside of the RTB exchanges and placing ghost bids on users in the control group eliminates noise. It also creates an apples-to-apples comparison between users in the treatment and control groups by considering all users that could have been exposed to an ad in a specific moment. Finally, this method allows marketers to test for incrementality at no additional cost.
Takeaways on Incrementality Testing
Proving the value of your paid growth campaigns continues to be one of the main challenges app marketers face today. Growth marketers can use incrementality testing to evaluate the individual value of their retargeting campaigns.
What is incrementality?
Incrementality in marketing measures the incremental lift that advertising spend contributes to the overall conversion rate.
Why is incremental testing important?
Incremental lift analysis allows marketers to optimize their ad spend; better assess the value of growth partners; and leverage users at all points in their sales funnel.
How do I do incremental testing?
A key aspect of all incremental test methodologies is the isolation of a holdout or control group that does not see ads.
Types of Incremental Testing Methodologies
- Intent To Treat (ITT): this is the most basic test methodology and includes serving an ad to a treatment group and not serving it to a control group.
- PSAs: this method serves real ads to both groups. Although it gets rid of noisy data, it’s costly and does not provide a clear apples to apples comparison since public service ads are incomparable to brand-related ads.
- Ghost Ads: Ghost ads eliminate issues of noisy data and create a clean testing environment by serving invisible “ghost” ads to users in the control group. However, the ghost ad and its bid data are visible to advertising partners. This gives information on how users who should have seen an ad behave. While ghost ads mitigate issues of noisy data and an apples-to-oranges comparison, they don’t work well for retargeting.
- Ghost Bids: This method is the most precise and cost-effective incrementality testing method when it comes to retargeting campaigns. This methodology eliminates users outside of the RTB exchanges and places ghost bids on users in the control group. This decreases noise and creates an apples-to-apples comparison for retargeting campaigns.
Take Action on Incrementality in Marketing Your Mobile App
Do you find yourself running out of strategies to re-engage mobile app users? An incremental lift analysis may be a great way to improve your re-engagement campaign when it comes to your mobile strategy. Ready to learn more or get help from the experts? YouAppi can help, send us a message here!
