As growth marketers, every decision must be data-driven and yield a strong return on investment. In order to be a successful marketer, it’s important to demonstrate validity of the information that is presented to teams and especially to clients. Incrementality is a metric for understanding the value of retargeting campaigns within the overall mobile marketing strategy. But how can you ensure your incremental results will continue to drive value? Enter: statistical significance. 

Statistical significance in marketing is a probabilistic indication of whether or not campaign results would have likely occurred even in the absence of the campaign. It’s important that results be statistically significant because marketers do not want to waste money on campaigns that won’t bring desired results. Marketers often run statistical significance tests within the first month of launching campaigns to test if specific variables are more successful at driving results than others.

Not a statistics expert? Don’t worry. We break down the value of statistical significance, how to calculate it and why it’s important to retargeting campaigns. 

What is Statistical Significance?

Statistical significance in marketing, at the most basic level, proves the relationships between the variables you're testing aren't random – meaning, that they influence one another. When it comes to retargeting campaigns, statistical significance is the parameter that indicates a user’s actions were a direct result of a campaign, or if similar results might have been observed even had the campaign never been run. 

When a campaign’s calculated uplift (the difference in response rate between a treated group and a randomized control group) is determined to be statistically significant, there exists strong evidence that the campaign was responsible for a user taking a desired action. This means the campaign is working!

It is important to understand statistical significance in marketing so that the results of campaigns can be replicated when you conduct another in the future. There is no point in spending valuable marketing hours and budget on campaigns that don’t show evidence of being the cause of the results shown. 

Calculating Statistical Significance

In order to measure a campaign’s statistical significance, marketers must conduct an incrementality test, define their hypothesis and have an understanding of their determined significance levels. Below are the steps necessary to complete this test so that you can prove your campaign’s value. 

1. Define the Hypothesis 

A hypothesis is an educated guess that can be proven wrong. There are two types of hypotheses - the null hypothesis and the alternative.

The null hypothesis assumes the difference between test variants in an A/B test is null, or amounts to nothing at all. In other words, there’s no conversion difference between test versions. 

Example: There is no difference in conversion results whether we serve users retargeted ads versus serving them nothing. 

The alternative hypothesis is quite literally, the opposite of the null hypothesis. Proving the null hypothesis wrong means that we find results to be statistically significant. As data-driven marketers, the goal is to prove campaign results reject the null hypothesis and are, therefore, statistically significant. 

Example: There are significantly higher conversion rates when we serve users ads versus not serving them ads. 

2. Collecting the Data

Determining a proper sample size is the first step in an experiment. In order to create a valuable test, the sample of users in the experiment needs to be large enough. The more data points contained in an experiment, the more reliable the analysis. From here, you can start to decide what groups you’d like to test and how long you want to run your campaign. 

3. Determine Significance Level (α)

Significance, or alpha (α), helps measure the risk of a Type I error. A Type I error is also known as a false positive and occurs when we believe there is evidence that the campaign was responsible for the behavior change within the test groups, but it actually wasn’t. Avoiding Type I errors is extremely important because if marketers determine a campaign as a “winner” when it’s not, revenue or conversions may suffer. 

The most commonly accepted risk level is .05 – meaning there is a 5% chance of a Type I error. The closer the significance is to zero, the lower the probability of Type I error and the more likely the results are accurate.

4. Begin the Incrementality Test

Now that you have determined sample size and timelines, it’s time to start incrementality testing. The incrementality of a retargeting campaign measures the additive lift that the spend invested in re-engaging users contributed to your overall goal. 

Incrementality testing starts with the random selection of a test group and a control group. Typically experimenters, and at YouAppi, will hold out 10% of users for the control group, leaving the remaining 90% in the test group. The test group receives the ad and the control group does not. 

After the test has been completed (usually at the 30 day mark), you can analyze the results and calculate the lift for each group in the experiment. At this stage, you may see positive results for both groups – this is where the statistical significance in marketing element comes into play.

Statistical Sig Graphic

5. p-Value

In order to find statistical significance, the p-value needs to be calculated. P-value is the probability of the test producing an observed result at the level or greater  than the results observed in your data — assuming the null hypothesis is true. In other words, a smaller p-value means that there is stronger evidence in favor of the alternative hypothesis.

↓ p-value → ↑ significant the finding

P-value is closely tied to the significance (α). When the p-value is less than α (p ≤ α), it means that we have proven the null hypothesis to be incorrect. As the significance value is typically set at .05, we would determine our test is statistically significant in marketing if the p-value is less than .05. 

In more scientific terms, a test would achieve statistical significance based on significance level (α), expressed as a p-value of ≤0.05 at a 95% level of confidence that you can reject the null hypothesis and don’t have a Type I error. 

Finding the p-value brings us to the end of this experiment and gives us the information necessary to confirm whether we have achieved statistical significance or if our campaign’s results are not accurate. When we discover that the campaign is statistically significant, you can confidently move forward with the process used to yield these results, knowing that you will deliver a strong ROI for your clients.

Takeaways of Calculating Statistical Significance in Marketing

In order to be a successful marketer, it’s important to demonstrate validity of the information that is presented to teams and especially to clients. While incrementality is a metric for understanding the value of retargeting campaigns, statistical significance indicates whether or not the results will continue to occur. Statistical significance, at the most basic level, proves the relationships between the variables you're testing aren't random – meaning, that they influence one another

We outline the steps necessary to complete this test so that you can prove your campaign’s value here - 

  • Define the Hypothesis: A hypothesis is an educated guess that can be proven wrong. There are two types of hypotheses – the null hypothesis and the alternative. Proving the null hypothesis wrong means that we find results to be statistically significant. 
  • Collect the Data: In order to create a valuable test, the sample of users in the experiment needs to be large enough. The more data points contained in an experiment, the more reliable the analysis.
  • Determine Significance Level (α): Significance, or alpha (α), helps measure the risk of a Type I error, also known as a false positive. Avoiding Type I errors is extremely important because if marketers determine a campaign as a “winner” when it’s not, revenue or conversions may suffer. The most commonly accepted risk level is .05 and the closer significance is to zero, the more likely the results are going to be accurate.
  • Begin the Incrementality Test: The incrementality of a retargeting campaign measures the additive lift that the spend invested in re-engaging users contributed to your overall goal. After this test is completed, you can analyze the results and calculate the lift for each group in the experiment. 
  • p-Value: The last step in finding the statistical significance in marketing is finding the p-value. The p-value is the probability of the test producing an observed result at the level or greater than the results observed in your data — assuming the null hypothesis is true. In other words, a smaller p-value means that there is stronger evidence in favor of the alternative hypothesis.