)
Incrementality testing methods for accuracy and success
){kind=small}
Incrementality testing serves as a "detective" for marketers, revealing the growth that can be directly attributed to marketing efforts beyond organic brand engagement or other external factors like seasonality. It isolates the true impact of a marketing activity—be it a campaign, tactic, or channel—removing guesswork and allowing marketers to identify what works and what doesn’t.
However, there are different methods of incremental analysis available to marketers, each of which has its own pros and cons. These methods are often packaged as a one-size-fits-all incrementality product.
Mehmet Emin Kazanç Senior Data Analyst at Adjust explains, “Not all incrementality testing methods are easy to implement and accurate. Geo-lift experiments, for example, often fail to account for complex influences that introduce noise and make it harder to isolate the true impact of a campaign. More sophisticated methods, such as causal inference, offer a clearer and more reliable picture by better isolating the true marketing impact, making them far more effective in today’s complex, multi-partner advertising environments.”
So, what exactly should you be looking for when choosing the right solution for your needs? Let’s dissect the different types of incremental analysis to understand their nuances.
Common incrementality testing methods
Let’s break down the most common methods for conducting incremental analysis, highlighting the pros and cons, so you can choose the right one for your specific needs.
)
A/B testing
A/B testing involves randomly assigning users into a control group (not exposed to ads) and a treatment group (exposed to ads), then comparing outcomes.
Pros:
- Simple and easy to understand.
- Effective for isolating the direct impact of a campaign.
Cons:
- May not account for external variables (e.g., seasonality, other campaigns).
- Requires a clear division of users, which can be difficult for smaller apps.
Geo-lifting
Geo-lifting Involves dividing geographical regions into test and control groups to measure the impact of marketing efforts in specific areas.
Pros:
- Can be implemented quickly for large-scale campaigns.
- Useful when user-level tracking isn’t possible.
- Applicable to region-specific sales strategies in industries like retail, telecommunications, etc.
Cons:
- It requires a very heavy lift upfront to implement, as a large amount of detailed knowledge, including demographics, market size, prior market performance, and more, is needed for each region.
- External factors (e.g., regional trends, competitors, etc.) can distort results.
- Less accurate in isolating the true effect of marketing, especially when factors like organic growth are not considered.
- Despite the significant resource investment required, the outcome is often limited to broad insights with reduced accuracy.
)
Melis Han
Senior Data Analyst, Adjust
Budget holdout
A portion of the marketing budget is withheld from certain regions or groups to act as a control, while the rest receive the full campaign.
Pros:
- Allows for a clear understanding of the impact of increased or decreased marketing spend.
Cons:
- Requires significant resources to run efficiently.
- Results may be skewed by other concurrent campaigns.
Causal inference
Uses statistical techniques to isolate the causal impact of marketing by comparing treated and control groups through synthetic control groups or other models.
Pros:
- Provides clear, accurate causal relationships.
- Accounts for external factors, making it highly reliable.
Cons:
- Complex to implement and often requires the support of advanced data science teams.*
- Can be costly due to the resources needed to set up and analyze results.
)
Mehmet Emin Kazanç
Senior Data Analyst, Adjust
*With InSight, Adjust’s next-gen incrementality solution, we’ve done the heavy lifting for you, so you can get all of the benefits of causal inference incrementality analysis without the need for a dedicated data science team.
This means we’ve done the groundwork of capturing vast amounts of data from multiple sources, sanitizing it by removing outliers, and meticulously organizing it for analysis. We also run complex scripts to analyze all historical data, ensuring accuracy across time. Additionally, we create synthetic control groups that are large enough and closely matched to your audience to eliminate data noise, providing you with precise insights without the need for a dedicated data science team.
)
The power of picking the right approach for long-term success
Choosing the right approach to incrementality testing depends on several factors:
Campaign type(s): Complex cross-channel campaigns might require a more sophisticated method like causal inference, while smaller, single-channel campaigns could benefit from simpler A/B testing if you're looking for a DIY in-house option. However, the low precision and inconsistencies often found in traditional A/B testing are still an important factor when making a decision.
Level of insights needed: If your goal is to get highly granular insights, advanced methods like causal inference can provide more accuracy. On the other hand, basic tests may suffice if you're looking for more generalized insights.
Budget: Some incrementality methods, such as geo-lifting, can be resource-intensive and costly, requiring significant ad spend. Others, like budget holdouts, are more affordable but may sacrifice some accuracy.
Precision vs. speed: If speed is your priority, methods like geo-lifting might give you faster results but at the cost of precision. If accuracy is more critical, causal inference may be worth the extra time and investment.
Linnéa Ghosh
Digital Marketing Analyst, Sleep Cycle
Whether you’re fine-tuning user acquisition strategy or maximizing your marketing budget, the ability to accurately measure the incremental impact of your efforts is crucial. Incremental analysis doesn’t just improve short-term performance; it also provides long-term benefits. Marketers can use it to avoid cannibalizing organic traffic and ensure that their paid efforts are generating truly beneficial results. But keep in mind–choosing the right method is critical!
Interested in harnessing the power of incrementality testing for your app? Get a demo to see how this method can work hand in hand with your attribution strategy to secure scalable growth.
Be the first to know. Subscribe for monthly app insights.
Keep reading
