> For the complete documentation index, see [llms.txt](https://docs.bird.com/applications/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bird.com/applications/campaigns/campaigns/concepts/holdout-groups.md).

# Holdout Groups

### Overview

Holdout groups, also known as control groups, are groups of your contacts that are intentionally excluded from receiving campaign messages. This allows you to measure the true impact of your campaigns by comparing the behavior of recipients against those who didn't receive your messages.

Bird offers two types of holdout groups: **Global Holdouts** that apply across all campaigns and **Campaign Holdout** for specific campaign testing.

### Why Use Holdout Groups?

Holdout groups help you answer critical questions about campaign effectiveness:

* What is the incremental impact of your messaging on conversions?
* Are your campaigns driving real behavior change or just reaching customers who would have converted anyway?
* What is the true ROI of your marketing spend?
* How do different campaign strategies compare in driving actual results?

### Global Holdout Groups

Global holdout groups exclude a group of contacts (based on a list or an attribute) from **all campaigns**, allowing you to measure the overall impact of your marketing efforts.

#### When to Use Global Holdouts

Use global holdouts when you want to:

* Measure the cumulative impact of all your marketing communications
* Understand baseline customer behavior without any marketing influence
* Calculate the overall incrementality of your marketing program
* Comply with customer preferences for reduced messaging frequency

#### Setting Up a Global Holdout

* Navigate to **Settings > Marketing > Analytics > Global Holdout**

<figure><img src="/files/MTH9HlfjuagHxdGZZXz1" alt=""><figcaption></figcaption></figure>

* Configure your holdout settings:
  * Select a list
  * Select a boolean (toggle) type attribute which when TRUE would make the contact part of Global Holdout group.
* Click on **Save**

<figure><img src="/files/NjMYcDQEYNpKlTsHFiJk" alt=""><figcaption></figcaption></figure>

### Campaign-Level Holdout Groups

Campaign-level holdouts exclude a percentage of recipients from a specific campaign, allowing you to test the impact of individual campaigns.

#### When to Use Campaign-Level Holdouts

Use campaign-level holdouts to:

* Test the effectiveness of a specific promotion or announcement
* Measure incremental impact of different messaging strategies
* Validate whether a campaign drives meaningful behavior change
* A/B test campaign presence vs. absence (not just message variants)

#### Setting Up a Campaign Holdout

* Create or edit your campaign in the Campaign Builder
* Navigate to the **Recipients** section > Advanced settings
* Enable **Holdout Percentage**

<figure><img src="/files/yl3KhSVD0ruAXc4UOUfg" alt=""><figcaption></figcaption></figure>

* Configure holdout settings as **Holdout percentage**: Typically 10-20% for campaign-level tests

<figure><img src="/files/JlufJZSOeNu8MFsGpX3n" alt=""><figcaption></figcaption></figure>

* **Randomization**: Bird automatically creates random assignment
* Launch your campaign

The holdout group will be excluded from this campaign only and can receive other campaigns normally.

#### Analyzing Holdout Results

Currently, Bird doesn't support a ready report for Global or Campaign Holdout analysis.

* You can view Global Holdout and Percentage Holdout metric in Campaigns Report

<figure><img src="/files/Y8HmqW0TYiclC5KToXfB" alt=""><figcaption></figcaption></figure>

* You can also view it in Campaigns list view and Export to csv

<figure><img src="/files/8ro7JgteOiyQWGqypp4v" alt="" width="375"><figcaption></figcaption></figure>

* You can find the contacts in segmentation who were skipped due to global holdout and export from Contacts page as needed
  * **Something was done or not done** > Select event = **Skipped Message Sending** at least once in the last x days
    * where **Reason equals global-holdout or campaign-holdout**

<figure><img src="/files/WrWMk7t3FEOmFGJluBf8" alt=""><figcaption></figcaption></figure>

You will need to export this data from Bird for offline analysis as follows:

**Calculating Incrementality:**

```
Incrementality = (Exposed Group Conversion Rate - Holdout Group Conversion Rate) / Holdout Group Conversion Rate × 100
```

**Example**: If exposed contacts convert at 8% and holdout contacts at 6%, your incrementality is 33.3%.

### Combining Global and Campaign Holdouts

You can use both holdout types simultaneously, but be aware of how they interact:

* **Global holdout**: Excludes contacts from all campaigns
* **Campaign holdout**: Excludes additional contacts from specific campaigns after global holdout exclusion
* **Total exclusion**: The sum of both holdouts reduces your reach


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.bird.com/applications/campaigns/campaigns/concepts/holdout-groups.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.