Holdout Groups
Overview
Holdout groups, also known as control groups, are groups of your contacts that are intentionally excluded from receiving journey messages. This allows you to measure the true impact of your campaigns by comparing the behavior of recipients against those who didn't receive your messages.
Bird offers two types of holdout groups: Global Holdouts that apply across all journeys and Journey Holdout for specific journey testing.
Why Use Holdout Groups?
Holdout groups help you answer critical questions about marketing effectiveness:
What is the incremental impact of your messaging on conversions?
Are your campaigns driving real behavior change or just reaching customers who would have converted anyway?
What is the true ROI of your marketing spend?
How do different campaign strategies compare in driving actual results?
Global Holdout Groups
Global holdout groups exclude a group of contacts (based on a list or an attribute) from all journeys, allowing you to measure the overall impact of your marketing efforts.
When to Use Global Holdouts
Use global holdouts when you want to:
Measure the cumulative impact of all your marketing communications
Understand baseline customer behavior without any marketing influence
Calculate the overall incrementality of your marketing program
Comply with customer preferences for reduced messaging frequency
Setting Up a Global Holdout
Navigate to Settings > Marketing > Analytics > Global Holdout
Configure your holdout settings:
Select a list
Select a boolean (toggle) type attribute which when TRUE would make the contact part of Global Holdout group.
Click on Save
Journey Step-Level Holdout Groups
Journey-level holdouts exclude a percentage of recipients from a specific campaign, allowing you to test the impact of individual campaigns.
When to Use Journey-Level Holdouts
Use campaign-level holdouts to:
Test the effectiveness of a specific promotion or announcement
Measure incremental impact of different messaging strategies
Validate whether a campaign drives meaningful behavior change
A/B test campaign presence vs. absence (not just message variants)
Setting Up a Journey Holdout
Go to Advanced Tab of a Send Message Step
You can disable Global holdout for a message if needed
Enable Holdout Percentage
You can select to apply the same journey holdout group as defined on the trigger to all the steps of the journey
You can define a step-level journey holdout group which applies to only this message.
Randomization: Bird automatically creates random assignment based on the percentage
The holdout group will be excluded from the specific journey steps only and can receive other journey messages normally.
Analyzing Holdout Results
Currently, Bird doesn't support a ready report for Global or Journey Holdout analysis.
You can view Global Holdout and Percentage Holdout metric in Journey Insights for a specific message step
You can view Global Holdout and Percentage Holdout metric in Journey Report
You can also view it in Journeys list view and Export to csv
You can find the contacts in segmentation who were skipped due to global holdout and export from Contacts page as needed
Something was done or not done > Select event = Skipped Message Sending at least once in the last x days
where Reason equals global-holdout or journey-holdout
You will need to export this data from Bird for offline analysis as follows:
Calculating Incrementality:
Example: If exposed contacts convert at 8% and holdout contacts at 6%, your incrementality is 33.3%.
Combining Global and Campaign Holdouts
You can use both holdout types simultaneously, but be aware of how they interact:
Global holdout: Excludes contacts from all campaigns
Journey holdout: Excludes additional contacts from specific journey steps after global holdout exclusion
Total exclusion: The sum of both holdouts reduces your reach
Last updated
Was this helpful?