Holdout Groups
Overview
Holdout groups, also known as control groups, are groups of your contacts that are intentionally excluded from receiving campaign messages. This allows you to measure the true impact of your campaigns by comparing the behavior of recipients against those who didn't receive your messages.
Bird offers two types of holdout groups: Global Holdouts that apply across all campaigns and Campaign Holdout for specific campaign testing.
Why Use Holdout Groups?
Holdout groups help you answer critical questions about campaign effectiveness:
What is the incremental impact of your messaging on conversions?
Are your campaigns driving real behavior change or just reaching customers who would have converted anyway?
What is the true ROI of your marketing spend?
How do different campaign strategies compare in driving actual results?
Global Holdout Groups
Global holdout groups exclude a group of contacts (based on a list or an attribute) from all campaigns, allowing you to measure the overall impact of your marketing efforts.
When to Use Global Holdouts
Use global holdouts when you want to:
Measure the cumulative impact of all your marketing communications
Understand baseline customer behavior without any marketing influence
Calculate the overall incrementality of your marketing program
Comply with customer preferences for reduced messaging frequency
Setting Up a Global Holdout
Navigate to Settings > Marketing > Analytics > Global Holdout
Configure your holdout settings:
Select a list
Select a boolean (toggle) type attribute which when TRUE would make the contact part of Global Holdout group.
Click on Save
Campaign-Level Holdout Groups
Campaign-level holdouts exclude a percentage of recipients from a specific campaign, allowing you to test the impact of individual campaigns.
When to Use Campaign-Level Holdouts
Use campaign-level holdouts to:
Test the effectiveness of a specific promotion or announcement
Measure incremental impact of different messaging strategies
Validate whether a campaign drives meaningful behavior change
A/B test campaign presence vs. absence (not just message variants)
Setting Up a Campaign Holdout
Create or edit your campaign in the Campaign Builder
Navigate to the Recipients section > Advanced settings
Enable Holdout Percentage
Configure holdout settings as Holdout percentage: Typically 10-20% for campaign-level tests
Randomization: Bird automatically creates random assignment
Launch your campaign
The holdout group will be excluded from this campaign only and can receive other campaigns normally.
Analyzing Holdout Results
Currently, Bird doesn't support a ready report for Global or Campaign Holdout analysis.
You can view Global Holdout and Percentage Holdout metric in Campaigns Report
You can also view it in Campaigns list view and Export to csv
You can find the contacts in segmentation who were skipped due to global holdout and export from Contacts page as needed
Something was done or not done > Select event = Skipped Message Sending at least once in the last x days
where Reason equals global-holdout or campaign-holdout
You will need to export this data from Bird for offline analysis as follows:
Calculating Incrementality:
Example: If exposed contacts convert at 8% and holdout contacts at 6%, your incrementality is 33.3%.
Combining Global and Campaign Holdouts
You can use both holdout types simultaneously, but be aware of how they interact:
Global holdout: Excludes contacts from all campaigns
Campaign holdout: Excludes additional contacts from specific campaigns after global holdout exclusion
Total exclusion: The sum of both holdouts reduces your reach
Last updated
Was this helpful?