;
};
### What are Bandits?
Bandits, also known as multi-armed bandits, are a type of experiment where the traffic assigned to each variation changes during the experiment. This is in contrast to standard experiments, where the percentages of traffic assigned to variations are static during the experiment.
When your bandit is running, we recalculate the percent of traffic to allocate to each variation based on which variation (or arm) is performing better on a single **Decision Metric**.
By driving more traffic to the best-performing variations, we can both:
* Identify the best variation faster than standard experiments
* Expose fewer users to poor-performing variations, which can help reduce negative impacts on metrics
Bandits therefore are extremely powerful, but they come with trade-offs:
* They require a single **Decision Metric** to optimize towards
* They can be more complex to set up
* They can introduce certain biases in estimating the true effect of variations
* They can, in some cases, actually do worse at picking the best variation than traditional experimentation.
[Get started with our Bandit setup guide!](/bandits/config)
If you expect the best variation to differ across user segments rather than there being one global winner, see [Contextual Bandits](/bandits/contextual), which learn a separate set of weights per user context.
### When should I run a Bandit?
You should run a bandit when:
* You want to reduce the cost of experimentation
* You have a clear decision metric you are optimizing for
* You have many arms you want to test against one another, and care less about learning about user behavior on several metrics
### Why should I run a Bandit?
1. Bandits can **reduce the cost of experimentation**. By allocating larger percentages of traffic to better performing arms, you avoid driving traffic to losing variations.
2. Bandits can quickly determine which variations are underperforming and direct traffic to the top performers, allowing you to **identify the best variation more quickly** than a standard experiment. With ten variations, a bandit may be able to quickly determine that several are poor performers. Sending traffic to the higher performing variations improves your ability to distinguish those high performers.
### Why should I run a standard Experiment instead?
1. Bandits require a **single decision metric** to optimize towards. If you have multiple metrics you care about, or if you want to understand the effect of each variation on multiple metrics, a standard Experiment may be better. Furthermore, a Bandit is more powerful if the decision metric is easy to measure right after experiment exposure. If you have a long sales cycle, or if you care about long-term effects, a standard Experiment may be better.
2. Bandits are known to **suffer from a [few types of *bias*](https://arxiv.org/abs/1905.11397)**. Because you adjust variation weights, stop early, and focus on the winners, Bandits can suffer from a variety of positive and negative biases when estimating variation averages. Because of this, standard Experiments can perform better in producing less biased estimates of experiment effects. We work hard to address some of these sources of bias in Bandits (e.g. by [weighting results across update periods](#weighting-results-across-update-periods), since users who enter your experiment on different days behave differently and have had different amounts of time to convert), but it is still a limitation of the Bandit approach.
3. Bandits can even perform worse at selecting the best variation more quickly than traditional experiments in some cases, such as when there are only two arms.
In that case, a standard Experiment can evenly split traffic between the two arms, which is best for reducing variance and increasing precision of lift estimates.
4. If you are not using [Sticky Bucketing](/app/sticky-bucketing), Bandits will cause users to change variations over time, which can be jarring for users if the feature being tested is not ephemeral, or easy to switch on and off for a given user.
### Comparison of Bandits and Experiments
While Experiments and Bandits can be used in a host of overlapping cases, there are some rough guidelines for when to use each:
| Characteristic | **Standard Experiments** | **Bandits** |
| - | - | - |
| Goal | Obtaining accurate effects and learning about customer behavior | Reducing cost of experimentation when learning is less important than just shipping the best version |
| Number of Variations | Best for 2-4 | Best for 5+ |
| Multiple goal metrics | Yes | No |
| Changing Variation Weights | No | Yes |
| Consistent User Assignment | Yes | Yes (with [Sticky Bucketing](/app/sticky-bucketing)) |
### GrowthBook's Bandit implementation
GrowthBook's Bandit uses Thompson sampling, a Bayesian algorithm that balances *exploration* (trying out different variations) and *exploitation* (focusing on the best-performing variations).
Thompson sampling allocates traffic proportionally to the probability that an arm is best.
By sending some traffic to arms that are less likely to have the highest revenue, Thompson sampling *explores* all variations when searching for a winner.
Thompson sampling *exploits* information about which arms are likely to be best. Sending more traffic to larger revenue arms saves money.
Exploration vs exploitation is a fundamental tradeoff in adaptive experimentation. Proportional allocation in Thompson sampling nicely balances this tradeoff.
In practice, GrowthBook ensures that each variation has at least 1% of traffic, in case your user behavior changes over time.
### Weighting results across update periods
Changing variation weights can make identical variations look different. Suppose users take up to a week to convert, and two variations perform the same. After the first week, noise makes one of them look slightly better, so the bandit starts sending it more new users. A week later, a larger share of that variation's users arrived in the last few days and might not have converted yet. If GrowthBook pooled all users, that variation would look worse even though the two perform the same.
To avoid this, GrowthBook groups users into cohorts by the period in which they were first exposed, where a period is the time between two weight updates. Variation weights stay fixed within a period, so within a cohort, users in every variation arrived over the same days and have had the same amount of time to convert. GrowthBook computes each variation's mean within each cohort and then averages the cohort means, using the same weights for every variation: each cohort's share of all users in the bandit. Every variation then reflects the same mix of early and recent users, and the shift in traffic no longer makes identical variations look different.
Weighting does not fix everything. Users in recent cohorts are still inside their conversion window, so each weight update counts only the conversions they have made so far. This matters when a variation changes how quickly users convert. For example, an urgency message might make users buy sooner without making more of them buy. That variation looks better early on, so the bandit sends it more traffic even though its final conversion rate is no higher. This is why we recommend setting the [update cadence](/bandits/config#update-cadence) based on how long users naturally take to convert.
## FAQ
1. **Can I run a Bandit using the frequentist engine?**
No. Currently Bandits are available only under the Bayesian engine, where Thompson sampling is easy to employ.
2. **How can I translate my frequentist p-value threshold into a success criterion for the Bayesian engine?**
While the frequentist p-value and the Bayesian chance-to-win are very different concepts, they are often conceptualized similarly by their practitioners. One simple approach is to adjust your settings so that your Bayesian chance to win (default is 95%) is equal to 1 minus half of your p-value threshold.
3. **How can I use power analysis for a Bandit?**
GrowthBook power results should be conservative for your Bandit. In many settings, Bandit power will be higher. However power in Bandits is much more complicated to estimate.
4. **What happens to users in worse-performing variants?**
As the experiment progresses, worse-performing variations will receive progressively less traffic. New users will not be exposed to those worse-performing variations. The experience of returning users depends on whether you have set up and are using sticky bucketing; with sticky bucketing, users will continue to see the same variation they were originally assigned to, without sticky bucketing, users could be re-assigned when they return to the experiment.
5. **What’s the difference between multi-armed bandit and contextual bandit experiments?**
Multi-armed bandits optimize traffic allocation among variations based purely on overall performance ([the Decision Metric](/bandits/config#5-selecting-a-decision-metric)). They treat all users the same, without taking individual user characteristics into account.
In contrast, contextual bandits use user-specific attributes (like location, device type, or past behavior) to select variations, aiming to tailor the experience to each user.
While multi-armed bandits aim for a global "best" variation, contextual bandits work to find the best variation for each unique user context. GrowthBook supports both: read [how Contextual Bandits work](/bandits/contextual), then follow [Setting up a Contextual Bandit](/bandits/contextual-config) to launch one.