Designing, Running and Analyzing A/B Testing Experiments

Experimentation is a broad topic that could easily fill a 500+ page textbook. I have had multiple opportunities to design, run, and monitor both online and offline experiments for new product features as well as scaling interventions. The goal of this article is not to cover every necessary detail required to design and run a good experiment but to share some of the knowledge from mistakes I made and also offer theoretical notes on some experimentation topics.

First, we have to define the specific goal and the anticipated value of the experiment to the business in our initial experiment design phase of hypothesis design. This usually begins with an internal history audit, documentation review and looking at observational studies to ensure that we are duplicating past efforts or testing questions that existing data already answers. Clearly articulating the "why and how it benefits the business" provides the necessary grounding for all the next design, execution, and interpretation phases. Avoid the common temptation of running tests simply to "see what happens" with a feature/intervention without a clear end goal and business value. Therefore, the work done before the first users are assigned determines whether your results will be actionable or noise. The common way to structure the experimentation process for actionable insights involves 5 critical steps:

1. Define the Goal and the Hypothesis

2. Define the Metric

3. User Eligibility, Randomization, and Logging

4. Duration, Sample Size, and Trade-offs

5. Analysis and the Ship Decision

The above steps require critical thinking of the users or entities for which we are running the experiment, in that, we must understand the product flow deeply enough to anticipate how a change will affect user behavior and most importantly having the judgment to make sound decisions. Good experimenters don't just analyze but act as product partners that connect statistical results to business outcomes, and they also know when it is enough to ship and when it isn't. That's the difference between reporting what happened and actually driving what happens next. Let's now look at each of these steps in more details:

Define the Goal and the Hypothesis

In this phase, our goal is to answer what behavior or outcome we trying to change. The goal needs to be specific and tied to a specific change, it should be measurable or trackable with a metric tied to a user action, and should predicts whether the outcome will increase or decrease.

As an example, let's imagine you built a feature to help lab technicians verify COVID results. You might hypothesize that including a confidence score alongside each result helps technicians make faster and more accurate decisions, because it reduces the cognitive load of interpreting the PCR machine readings without additional context. That is a strong hypothesis because it is grounded in a real user pain point of large sequence machine readings, it predicts a behavioral shift, and it tells you exactly which direction the metric should move. To build one, start by identifying the user pain, think about what change would shift the experience, and ask yourself why this change would improve the experience and business value.

Alongside defining the goal and the hypothesis, we also need to specify what our treatment and control groups are in this phase. The treatment is the changed experience we are testing, and the control is the baseline we are comparing it against. Once both are defined, we move to the next phase of metrics.

Define the Metrics

The purpose of metrics is to help you understand how the experiment is performing and has performed. The primary metric, sometimes called the north star metric, is the success metric that reflects the main goal you are trying to influence. It is always good to identify 2 to 3 candidate metrics first, then narrow down to one based on the goal. The primary metric must be available for both the control and treatment groups. In our COVID example, the primary metric might be test verification rate, since the goal is to speed up results verification. Both the control and treatment subjects should be able for example to verify the test for the two groups to be comparable.

Guardrail metrics are what we aim to protect at all cost. We are not optimizing for them, but most importantly do not want them to get worse. Examples include application crash rate, page load times, etc. It is a good practice to have 2 to 5 of these metrics to monitor throughout the experiment. They act as a safety net for not shipping a winning experiment that causes chaos and degrades user experience.

Alongside guardrail metrics, we also need to define sanity checks and tracking metrics. These help in the verification that the change is working as expected, for example, that users can actually see or interact with the new feature you are testing, and that the backend is correctly logging movement in the metrics. If your sanity checks are failing, your experiment results cannot be trusted regardless of what the numbers say.

Finally, while defining your metrics, avoid vanity metrics that are not tied to the hypothesis being tested, and avoid choosing a primary metric that is too hard to shift within your experiment window and most importantly, do not rely on a single metric to evaluate the whole experiment.

Randomization and User Selection

Once the hypothesis and metrics are defined, the next step is making sure the right users are included through the right unit of randomization, the variant exposure is fair, we know when each unit will enter the experiment from the user funnel and we are avoiding network effects or any form of interference. There are three key pieces to get right: eligibility, randomization, and instrumentation.

Eligibility is about defining who qualifies for inclusion. In our COVID lab feature example, you would want to exclude users who would never encounter the treatment or reach that stage, lab technicians who have not yet been onboarded to the platform, or users accessing the system from unsupported devices. These users have zero chance of completing a test verification since they are unauthorized, including them would dilute the experiment. It might make sense to include users who only reach the final stage of the funnel being tested, so that we have the minimum dilution possibilities.

Randomization ensures unbiased assignment between groups. Usually the gold standard is user-level randomization, meaning for example a given lab technician will always sees the same experience throughout the experiment. This is what sticky assignment achieves once a user is placed in control or treatment, they stay there for the duration. Sometimes this rule breaks if we are running a multi-armed bandit experiment. Session-level randomization is sometimes used, but only in situations where the nature of the task does not carry persistent effects across sessions, which mostly in a lab diagnostic workflow like ours is rarely the case. At this stage we also need proper instrumentation and logging design to capture all the events for further analyzing our outcomes.

Duration and Sample Size Calculation

Since the intial design choices are in place, how long should we run the experiment, and how many users or observations do we need to detect an effect? Before we can answer these questions, let's first ground ourselves with some foundations (there will be some maths but you can jump the equations), that is, statistical tests, p-values, confidence intervals, effect size, statistical power, and powwer analysus then we shall be ready to answer those questions.

1. Hypothesis Testing and Errors

Here, it is important to pause and distinguish between stating a hypothesis and testing it. While in Phase I we were stating an hypothesis, hypothesis testing is the process of evaluating that claim against observed evidence. We are now asking how much evidence does our data provide to reject it. Simply, hypothesis testing is a way two opposing claims are evaluates.

The null hypothesis ($H_0$) represents the status quo or the "no effect" default. We assume this statement is true unless the data proves otherwise. In contrast, the alternative hypothesis ($H_a$) is the actual claim we are trying to detect.

Our claim can be one-sided (predicting a specific direction e.g Feature A increases conversion rate i.e $H_0: \mu_A \le \mu_B$, $H_1: \mu_A > \mu_B$) or two-sided (predicting any difference e.g Feature A changes conversion rate that could increase or decrease i.e $H_0: \mu_A = \mu_B$, $H_1: \mu_A \ne \mu_B$). Notably, a hypothesis test is not designed to prove that $H_0$ is true. Instead, it measures whether the observed data is inconsistent enough with the null hypothesis to justify rejecting it in favor of $H_a$.

Each hypothesis concludes with a decision of either rejecting $H_0$ or failing to reject it. Because we rely on sample data rather than population we won't be able to observe, this decision can result in four outcomes which two are correct and two erroneous as shown below:

$$\begin{array}{lll} \textbf{Decision} & H_0 \text{ is True} & H_0 \text{ is False} \\ \hline \text{Fail to Reject } H_0 & \text{Correct decision} & \text{Type II Error } (\beta) \\ \text{Reject } H_0 & \text{Type I Error } (\alpha) & \text{Correct decision (Power)} \\ \end{array}$$

Our main goal is to control the False Positives. The Significance Level ($\alpha$) which then is our primary goal is to control the probability of a Type I error. This significance level ($\alpha$) represents our tolerance for false positives.

We define $\alpha$ formally as:

$$\alpha = P(\text{Reject } H_0 \mid H_0 \text{ is true}) \tag{1}$$

In most cases, the default is $\mathbf{\alpha = 0.05}$, meaning we accept a 5% risk of being wrong when claiming a result is significant. However, for higher-stakes environments like clinical trials, this is often $\mathbf{\alpha = 0.01}$.

With the significance level, the hypothesis testing process is summarized as follows:

1. Stating the Hypotheses: Clearly state what the null ($H_0$) and alternative ($H_a$) hypotheses using a specific population parameter. The null must be testable statement (e.g., $\mu = \mu_0$), and the alternative shows the direction of the change you aim to detect.

2. Calculate the test statistic: The way to do this depends on your data's distribution and the parameter being tested (In the next subsections we shall cover t-statistics, z-statistics, and quasi-experiments methods).

3. Evaluate the evidence via the p-value: We Compare test statistic against the null distribution(the results we would expect if $H_0$ were actually true)

4. Make a Decision: We compare the p-value to our pre-set significance level ($\alpha$). If $p \leq \alpha$, we reject $H_0$. If $p > \alpha$, you fail to reject $H_0$.

It is important to remember that failing to reject $H_0$ is not the same as proving it is true. Not having the evidence against the null does not confirm the status quo; it may simply indicate that test had low statistical power or that the data was insufficient to detect a real effect. Hypothesis testing alone with p-values is not enough.

2. Power

The power of a test is the probability of correctly rejecting the null hypothesis ($H_0$) when the alternative hypothesis ($H_a$) is actually true. Unlike the significance level ($\alpha$), the power relates to the true parameter value ($\theta_1$). Formally, power quantifies the likelihood of your test detecting an effect that truly exists as:

$$Power(\theta_1) = P(\text{Reject } H_0 \mid \theta_1 \text{ is true}) = 1 - \beta(\theta_1) \tag{2}$$

The $\beta(\theta_1)$ represents the probability of a Type II error (failing to reject a false null). When the probability of missing a real effect decreases, the power of your test increases. In A/B testing, it is common to use a power of 0.8 i.e 80%.

3. Statistical tests

The test statistic is calculated value from the sample data that helps us make the decision on whether or not to reject the null hypothesis. It condenses observations into a format that can be compared against a reference distribution (or null distribution). This distribution represents the range of values the test statistic would typically take if the null hypothesis were actually true.

3.1 z-statistic and t-statistic

Given a population mean $\mu$, we use a test statistic that measures how far the observed sample mean $\bar{Y}$ deviates from the null hypothesis value $\mu_0$. When the population variance $\sigma^2$ is known, we standardize this difference by dividing it by the standard error, $\sqrt{\sigma^2/n}$ resulting in Z-statistic:

$$Z = \frac{\bar{Y} - \mu_0}{\sqrt{\sigma^2/n}} \tag{3}$$

Under the null hypothesis $H_0$, this statistic follows a standard normal distribution $N(0,1)$, which acts as our reference. Meaning, the z-test determine whether two population means are different when the variance is known. In practice, suppose we measure an outcome $Y$ (e.g., conversion rate or revenue per user). For two variants A and B, our population means would be

$$\mu_A = \mathbb{E}[Y_A], \quad \mu_B = \mathbb{E}[Y_B] \tag{4}$$

These represent the true expected outcomes for each variant. We would then estimate the metric by computing the sample means and the the difference in means as:

$$\bar{Y}_A = \frac{1}{n_A}\sum_{i=1}^{n_A} Y_{Ai} \tag{5}$$ $$\bar{Y}_B = \frac{1}{n_B}\sum_{i=1}^{n_B} Y_{Bi} \tag{6}$$ $$\Delta = \bar{Y}_B - \bar{Y}_A \tag{7}$$

But under the null hypothesis we assume no treatment effect $H_0: \mu_B - \mu_A = 0$ and $Z \sim N(0,1)$. We would now calculate our test statistic with the population variances with the Z-statistic as:

$$Z = \frac{\bar{Y}_B - \bar{Y}_A} {\sqrt{\frac{\sigma_A^2}{n_A} + \frac{\sigma_B^2}{n_B}}} \tag{8}$$

The most important thing is that, for our Z-statistic, we comparing against normal distribution $N(0,1)$ because that is the distribution we expect it to follow when the null hypothesis is true.

However, when we no longer know the population variance $\sigma^2$, the t-statistic come in play. We use the sample variance $s^2$ which is itself almost random and sometimes fall below the true $\sigma^2$ resulting into additional variability into the test statistic. Our null distribution must therefore be more accommodating of extreme values than the standard normal, which the t-distribution provides.

The t-distribution, denoted $t(\nu)$ closely resembles the standard normal in shape but its control by the degrees of freedom parameter, that results in heavier tails and a sharper peak. So, substituting $s^2$ for $\sigma^2$ in the z-statistic results in the t-statistic:

$$t(\mu_0) = \frac{\bar{Y} - \mu_0}{\sqrt{s^2/n}} \tag{9}$$

Before running an experiment, we have to select the appropriate statistical test for analysing the results. In most A/B testing scenarios, a t-test is enough. However, there are settings where the experimental design is more complex and assignment to treatment cannot be fully randomised, or basic assumptions of constant variance or independence are violated. In these cases, quasi-experimention methods such as regression discontinuity design, difference-in-differences, and instrumental variables become essential.

Furthermore, even within the t-test process, we need to know how to estimate the variability of the test statistic. The standard approach above relies on the standard error of the difference in means, defined as:

$$SE = \sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}} \tag{10}$$ $$t = \frac{(\bar{Y}_A - \bar{Y}_B)}{\sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}}} \tag{11}$$

This assumes that the variance of the outcome is stable across observations. When that assumption holds, the standard error is enough. When it does not, using this test statistics will either be too liberal or too conservative. Some of the alternatives to the standard error include:

1. Heteroskedasticity-Robust Standard Errors

The standard errors assume homoskedasticity meaning that the variance of the outcome is constant across all groups. In practice, a new feature might cause some users in the treatment group to act with much higher variability than the control group. A potential solution is the heteroskedasticity-robust standard errors defined as:

$$S E_{\text{robust}} = \sqrt{\frac{n}{n-k} \cdot \frac{1}{n^2} \sum_{i=1}^{n} \hat{e}_i^2 \cdot x_i^2} \tag{12}$$

This is useful when we suspect "outliers" or specific segments of users driving more variance than others. So, instead of assuming a shared average variance, we estimate the spread using each observation's own squared residual ($\hat{e}_i^2$). In some industries, it is a standard practice to use robust standard errors by default.

2. Clustered Standard Errors

When observations are not independent aka clustered, for example you randomize at the level of a city, device type, or time period, rather than the individual user. All users in a "cluster" likely behave similarly for reasons unrelated to your treatment which might artificially inflates your samples. We use clustered standard errors whenever your unit of randomization (e.g., a school) is at a higher level than your unit of measurement (e.g., a student):

$$SE_{\text{cluster}} = \sqrt{\frac{G}{G-1} \cdot \frac{n-1}{n-k} \cdot \frac{1}{n^2} \sum_{g=1}^{G} \left( \sum_{i \in g} \hat{e}_{ig} \cdot x_{ig} \right)^2} \tag{13}$$

Other options that are less used include Bootstrap Standard Errors, Newey-West Standard Errors, among others.

3.2 Quasi-experiments

Difference in Difference(DiD)

When randomization is impossible, with a treatment and control group across two timelines pre-intervention and post-intervention, instead of simply comparing the groups after the treatment, DiD isolates the treatment effect by subtracting the baseline difference that already existed between them. That is, we measure the change in the treatment group and subtracts the change in the control group as:

$$\hat{\delta}_{\text{DiD}} = (\bar{Y}_{\text{treat, post}} - \bar{Y}_{\text{treat, pre}}) - (\bar{Y}_{\text{control, post}} - \bar{Y}_{\text{control, pre}}) \tag{14}$$

However, real-world problems in most cases are rarely linear. Metrics such as lab test orders (during covid), revenue, or conversion rates are subject to seasonality and other macroeconomic fluctuations that a simple two-period model cannot account for. Commonly a two-way fixed effects extension is used, where unit fixed effects $\alpha_i$ and time fixed effects $\lambda_t$ replace the group and period indicators as:

$$Y_{igt} = \alpha_i + \lambda_t + \delta \cdot (\text{Treat}_g \times \text{Post}_t) + X_{it}\beta + \varepsilon_{igt} \tag{15}$$

$X_{it}$ is a vector of additional covariates that control for variables and external factors like trend and user characteristics. Also, $\alpha_i$ takes into consideration the time-invariant heterogeneity across units, while the time fixed effects $\lambda_t$ accounts for shocks to all units in a given period.

The validity of DiD is due to the parallel trends assumption that states, the treatment and control groups would have followed the same trajectory over time as:

$$\mathbb{E}[Y_{it}(0) - Y_{it-1}(0) \mid \text{Treat}_i = 1] = \mathbb{E}[Y_{it}(0) - Y_{it-1}(0) \mid \text{Treat}_i = 0] \tag{16}$$

Since this assumption is untestable post-treatment, it is usually assessed empirically with a test of equivalence before finalising treatment and control assignments which is essentially an A/A test.

Other common quasi-experimental methods for A/B testing include Regression Discontinuity Design (RDD) and Instrumental variable (IV).

4. P-values

A p-value is the probability under the null hypothesis of observing a test statistic at least as extreme as the one observed from the data. Formally, for a two-sided test:

$$p = P(|Z| \geq |z_{\text{obs}}| \mid H_0 \text{ is true}) \tag{17}$$

Its worth pausing since p-values are frequently misunderstood. The p-value is not the probability that the null hypothesis is true. These two statements are not the same thing, and mixing them leads will eventually led to overconfident misleading conclusions. For example, let's assume we are testing $H_0: \mu = 30$ at significance level $\alpha = 0.05$ and obtain a p-value of $0.05$. The correct interpretation is, If the true mean really is 30, there is a 5% chance we would see a result this far from the it just by luck. It is not a statement that there is a 5% chance the mean is truly 30 and a p-value says nothing about that probability directly. Summarize it as:

$$p = P(|Z(\mu_0)| \geq |z_{\text{obs}}| \mid \mu = \mu_0) \neq P(\mu = \mu_0 \mid \text{data}) \tag{18}$$

5. Confidence intervals (CI)

CI provide a range of plausible values for an unknown population parameter based on observed sample data. Instead of reporting only a single point estimate (such as a sample mean), a confidence interval quantifies the uncertainty around that estimate. For example, a 95% confidence interval represents a procedure that produces intervals containing the true parameter in 95% repeated samples.

Given a population variance, we use the Z statistic with a rejection region for a two-sided hypothesis test as:

$$|Z(\mu_0)| > z_{1-\alpha/2} \tag{19}$$

Since $Z(\mu_0) = \frac{\bar{Y}-\mu_0}{\sigma/\sqrt{n}}$, the rejection condition could be rearranged to find all values of $\mu_0$ that would not be rejected as:

$$\left|\frac{\bar{Y}-\mu_0}{\sigma/\sqrt{n}}\right| < z_{1-\alpha/2} \tag{20}$$ $$-z_{1-\alpha/2} < \frac{\bar{Y}-\mu_0}{\sigma/\sqrt{n}} < z_{1-\alpha/2} \tag{21}$$

Solving for $\mu_0$ would finally result in (1 − $\alpha$) confidence interval:

$$\left( \bar{Y} - z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}}, \; \bar{Y} + z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} \right) \tag{22}$$

However, if the population variance is unknown (which is the common case), we replace $\sigma$ with a sample standard deviation and use a t-distribution $n-1$ degrees of freedom. Our CI result in:

$$\left( \bar{Y} - t_{1-\alpha/2,\,n-1}\frac{s}{\sqrt{n}}, \; \bar{Y} + t_{1-\alpha/2,\,n-1}\frac{s}{\sqrt{n}} \right) \tag{23}$$

Confidence intervals are also widely misunderstood. The most common misinterpretation is that “There is a 95% probability that the true mean lies inside CI values”. This is a gross misinterpretation because, in the frequentist thinking, the population parameter is fixed and does not vary randomly.

The correct interpretation is If we repeated the same sampling process many times and constructed a confidence interval each time, 95% of those intervals would contain the true population mean. So, the focus is on the performance of interval generation procedure, not the probability that the parameter lies inside a particular interval.

6. Effect size

As we have seen with the p-values and CIs, statistical significance alone does not show whether an observed difference is meaningful. With large sample sizes, we can generally produce lower p-value, but to better understand the magnitude of an effect, we use the effect size. Effect size quantifies the difference between two groups relative to the variability in the data. Cohen's d is commonly used to standardized effect size of two means as:

$$d = \frac{\bar{Y}_T - \bar{Y}_C}{s_p} \tag{24}$$

$\bar{Y}_T$ represents the treatment mean, $\bar{Y}_C$ the control mean and $s_p$ is the standard deviation across groups. The Common Effect Size classification is as shown below:

$$\begin{array}{ll} \textbf{Magnitude} & \textbf{Approximate Value} \\ \hline \text{Very Small} & 0.01 \\ \text{Small} & 0.2 \\ \text{Medium} & 0.5 \\ \text{Large} & 0.8 \\ \text{Very Large} & 1.2 \\ \end{array}$$

In addition to standardized measures, we can report the raw differences between group averages which usually provide the actual magnitude of the difference in the metric of values between the two groups as:

$$\Delta = \bar{Y}_T - \bar{Y}_C \tag{25}$$

Both standardized effect sizes and raw differences are typically reported together in modern experimentation analyses.

Power Analysis

We now have some good theoritical context, let's revisit our two questions; how many observations are required, and how long should the experiment run?. This is what power analysis will help us answer with either sequential design or fixed-sample design.

1. Fixed-Sample Design

In a classical A/B test, the sample size is calculated before the experiment begins and we runs the test until that sample size is reached. We start by setting the significance level $\alpha$, the target power $1 - \beta$, the variance $\sigma^2$, and the Minimum Detectable Effect (MDE) which is the smallest change we want our experiment to detect.

For a two-sample comparison of means with equal group sizes and known variance, the required sample size per group is computed as:

$$Z = \frac{\bar{Y}_T - \bar{Y}_C}{\sqrt{2\sigma^2/n}} \sim N\left(\frac{\delta}{\sqrt{2\sigma^2/n}},\ 1\right)$$ $$n = \frac{2\sigma^2 \left(z_{1-\alpha/2} + z_{1-\beta}\right)^2}{\delta^2} \tag{26}$$

For example using the default values $z_{0.975} \approx 1.96$ and $z_{0.8} \approx 0.84$ would result in the sample size of:

$$n \approx \frac{2\sigma^2 (2.80)^2}{\delta^2} = \frac{15.68\,\sigma^2}{\delta^2}$$

Given the required sample size, $n_{\text{daily}} = U \cdot p$, the required duration in days would be:

$$D = \frac{n}{U \cdot p} \tag{26}$$

For example, with $U = 100{,}000$ users per day, a 50/50 split, and a required sample size of $n = 350{,}000$ we would require to run this experiment for at least:

$$D = \frac{350{,}000}{50{,}000} = 7 \text{ days}$$

2. Sequential Experiment Design

The fixed sample method is great and easy, however, in practice we need to monitor an experiment continuously and stop early if we have a strong evidence either in favor of the alternative or against it. Naively peeking at the experiment and stopping when $p < 0.05$ might inflate the Type I error rate well above $\alpha$, because each look gives an additional opportunity to incorrectly reject $H_0$. Sequential designs solve this by defining decision boundaries that account for continuous monitoring.

At time $t$, after observing $n_t$ samples per group, the test statistic will be:

$$Z_t = \frac{\bar{Y}_T - \bar{Y}_C}{\sigma\sqrt{2/n_t}} \tag{27}$$

With Wald's SPRT, we monitors the likelihood ratio of the alternative to the null at each step:

$$\Lambda_t = \frac{L(H_1 \mid Y_1, \ldots, Y_t)}{L(H_0 \mid Y_1, \ldots, Y_t)} \tag{28}$$

We reject $H_0$ if $\Lambda_t \geq B$, accept $H_0$ if $\Lambda_t \leq A$, and continue sampling otherwise using the two decision thresholds defined in terms of the error rates:

$$A = \frac{\beta}{1 - \alpha}, \qquad B = \frac{1 - \beta}{\alpha} \tag{29}$$

Therefore, the required sample size under $H_1$ is:

$$\mathbb{E}[n \mid H_1] \approx \frac{\beta \log(A) + (1-\beta)\log(B)}{\mathbb{E}[\log \Lambda_1 \mid H_1]} \tag{30}$$

This approach is highly efficient for large treatment effects, and often saves time compared to fixed-sample tests.

Once we have a clear power analysis method to determine the required sample size and experiment duration, one of the final considerations in the experiment design phase is whether we are testing for multiple metrics simultaneously. This matters because every hypothesis test carries an $\alpha$ probability of a false positive. When we test $m$ hypotheses independently at significance level $\alpha$, the probability of making at least one Type I error which is the familywise error rate (FWER) will inflates drastically as:

$$P(\text{at least one false positive}) = 1 - (1-\alpha)^m \tag{31}$$

Some of the potential solutions are:

1. Bonferroni Correction

This is the most conservative but straightforward, tt controls the FWER by dividing the significance threshold equally across all $m$ tests:

$$\alpha_{\text{Bonferroni}} = \frac{\alpha}{m} \tag{32}$$

A test is declared significant only if its p-value satisfies $p_i \leq \alpha/m$. The FWER follows the union bound:

$$P\left(\bigcup_{i=1}^m \{p_i \leq \alpha/m\}\right) \leq \sum_{i=1}^m P\left(p_i \leq \frac{\alpha}{m}\right) = m \cdot \frac{\alpha}{m} = \alpha \tag{33}$$

2. False Discovery Rate

Instead of controlling the probability of any false positive, this controls the expected proportion of rejected hypotheses that are false positives. That is, if $V$ is the number of false rejections and $R$ is the total number of rejections, it results in:

$$\text{FDR} = \mathbb{E}\left[\frac{V}{\max(R, 1)}\right] \tag{34}$$

The Benjamini-Hochberg procedure is a common method to control FDR. It sorts the p-values in ascending order and finds the largest $k$ such that:

$$p_{(k)} \leq \frac{k}{m} \cdot \alpha \tag{35}$$

All hypotheses corresponding to $p_{(1)}, \ldots, p_{(k)}$ are rejected. This method is less conservative than Bonferroni and is more powerful when many tests are being conducted, but it allows for a controlled proportion of false positives.

Finally, once the experiment concludes running and the data is analyzed, our final decision depends on both statistical evidence and practical considerations from product partners(sometimes its not worth making a change even with detected effect). The treatment effect, confidence interval, and metric trade-offs are reviewed to determine whether the feature should be launched, modified, or discarded. It is a good practice for the experiment process and results to be documented and becomes part of an organizational knowledge base that informs future product development and improvements.