> For the complete documentation index, see [llms.txt](https://docs.bird.com/applications/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bird.com/applications/journeys/journeys/concepts/a-b-testing.md).

# A/B Testing

A/B testing in Journeys helps you find which message variation performs best so that majority of your audience receives the best-performing content. You can test different versions of a message within a **Send Message** step—such as changing the subject line or content—and automatically pick the winner based on performance.

### What You Can Do

With A/B testing in a **Send Message** step, you can:

* Create **multiple message variations** (A, B, C, etc. upto 8 variations)
* Define **traffic distribution** between variations
* Choose a **winning metric** (like opens, clicks, or conversions)
* Set a **time limit** for the test to run (default: 365 days)
* Use **AI-based optimization** to automatically shift traffic to better-performing variations
* View detailed **results and winner metrics** in reports
* **Manually end** the test or **select a winner** anytime

### Setting Up an A/B Test

#### Add a Send Message Step

In your Journey builder:

* Drag a **Send Message** step onto the canvas.
* Select the message channel and content (Email, Push, SMS). Note that it is currently not supported for WhatsApp.
* Click **Create A/B Test** inside the step.

<figure><img src="/files/ZffamlqKo8xcnzKybjXj" alt="" width="375"><figcaption></figcaption></figure>

#### Add Variations

You’ll start with Variation A and B when you create an A/B test.

* When you create an A/B Test it will be in **Draft** status.
* To remove A/B Test, simply click on the Delete icon.

<figure><img src="/files/ChomB6ch7AywxL52WODt" alt="" width="375"><figcaption></figcaption></figure>

To add more:

* Click **Duplicate** on any variation to C, D and so on.

<figure><img src="/files/iqd8tbpuvvPnx1DioBEU" alt="" width="375"><figcaption></figcaption></figure>

* You can click **Edit template** to update the content
* Each variation can be edited to have:
  * Different subject lines or headers
  * Different message body or creatives
  * Different CTA wording or tone
* You can remove a variation after 2 variations using **Remove** option

**Tip:** Test one key element at a time (e.g., subject line or CTA) for clearer results.

### Configure A/B Test Settings

Once you’ve added variations, set up how the test will run.

* Click on **A/B Test Settings** to navigate to the settings page

<figure><img src="/files/cait86PJ1FqyPmOorsqT" alt="" width="375"><figcaption></figcaption></figure>

#### Test Name

You can give a name to your test so that you can uniquely identify it from any other tests you may add it in other message steps in the same journey.

#### **Sending Distribution**

Define what percentage of your audience should receive each variation to begin with. By default, it is equally distributed between all variations.\
Example:

* A → 50%
* B → 50%

The system will automatically adjust the distribution to the better-performing variation after initial data is analyzed.

#### **Winning Metric**

Choose the performance metric that decides the winner:

* **Open Rate**
* **Click Rate** (Default)
* **Conversion Rate**

#### Automatic Winner Selection

The test will run till the selected winning metric reaches statistical significance to determine a winner with sufficient confidence or the defined time duration limit whichever comes earlier.

#### **Time Duration**

Set how long the test should run before it ends with or without winner determination. By default, it is set as 365 days.

If no clear winner is determined within this time, you can manually select one.

Your changes to test settings will be automatically saved. You can also edit or duplicate to add more variations in this page

Additionally, you can preview your content by clicking on the preview eye icon.

<figure><img src="/files/CE8FSCgKW43pu0qYgUOv" alt=""><figcaption></figcaption></figure>

### Publishing a Test

When you are done with A/B Test settings, you can then publish the journey which would publish any draft tests created in the journey

* Once published, the test will show In Progress status
* You can click on View report to see the performance details for the test

<figure><img src="/files/EOhzYhWJALTUSikLUTLH" alt="" width="375"><figcaption></figcaption></figure>

* You can manually end the test at any time, by clicking on **A/B Test Settings > End Test**

<figure><img src="/files/151KeRtFpQEq8gyb6g4D" alt=""><figcaption></figcaption></figure>

### AI-Powered Optimization

Journeys uses an **AI-driven multi-armed bandit algorithm** to automatically optimize traffic distribution among variations in real time.

Instead of splitting audience traffic evenly for the entire test duration, the algorithm continuously **shifts more traffic to the better-performing variations** based on the chosen winning metric (e.g., click rate).

Here’s how it works:

#### **When AI Optimization Begins**

* AI optimization starts **after each variation has at least 50 delivered messages**.
* The test must have been running for **at least 2 days** to ensure there’s enough data for confident evaluation.

#### **How Optimization Works**

* Once activated, the algorithm dynamically updates traffic allocation—sending a higher share of new audience members to variations showing stronger performance.
* This approach allows your campaign to **maximize engagement even while testing is in progress**.

#### **When the Test Ends Automatically**

* The AI continuously monitors performance.
* The test **ends automatically** when:
  * The algorithm determines that **no further optimization is possible**, and
  * A **clear winner** is identified based on the **confidence level of performance** differences.
  * The **time duration limit** is reached.

If the algorithm finds that performance differences are **not statistically significant**, no automatic winner will be chosen.

The algorithm needs at least 95% confidence in 100 iterations of traffic distribution changes. It currently evaluates the performance and performs adjustments to distribution every 30 mins.

The maximum percentage change of traffic for each variation can be 20%. When the experiment reaches a stage where the traffic distribution cannot be more than 1% and the confidence reaches 95%, the test is completed.

### Selecting a Winner

You can **manually end a test** at any time by selecting **“End Test and Pick Winner.”**

There are three possible outcomes:

**Winner Automatically Selected by AI**

* The AI identifies a clear top performer and assigns the remaining traffic to that variation.
* The test ends automatically, and all future contacts entering this step receive the winning version.

**Manual Winner Selection**

* If the test ends without sufficient confidence or the test is still running, you can manually review performance metrics and choose a winner.

<figure><img src="/files/NaCnOb8mqE6NF0SbX1cZ" alt="" width="375"><figcaption></figcaption></figure>

**No Winner Determined**

* In cases where the data is inconclusive (low volume, small differences, or short duration), no winner will be automatically picked.
* You can still manually select a variation based on observed performance.

<figure><img src="/files/zJO6LY18TXlmjS44Es9A" alt="" width="375"><figcaption></figcaption></figure>

### Viewing Results

All test metrics and outcomes are available in the **A/B Test** tab on the **Journey Report page**.

Here you can see:

* Each variation’s performance based on winning metric
* Number of sent and deliveries per variation
* Variation performance changes over time (for AI-optimized tests)
* Selected or auto-selected winner

<figure><img src="/files/A0I4Zd3uqcAXh0ZCZfmh" alt=""><figcaption></figcaption></figure>

You can also view the metrics filtered by variation in the **Journey Insights** view by opening a **Send Message** step.

<figure><img src="/files/WJyvhU2KnEd7sV5Oi6ma" alt=""><figcaption></figcaption></figure>

### Best Practices

* Test only **one element at a time** for clear insights.
* Allow at least **2–3 days** for sufficient data before evaluating results.
* Ensure each variation has **enough recipients (≥50)** to enable AI optimization.
* Don’t end the test too early — confidence builds over time.
* Use results to refine future message content and improve journey performance.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.bird.com/applications/journeys/journeys/concepts/a-b-testing.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.