--- title: "Sequential One-Way ANOVA" author: "Meike Snijder-Steinhilber" date: "`r Sys.Date()`" output: rmarkdown::html_vignette: toc: true toc_depth: 4 description: > This vignette describes the sequential one-way ANOVA. vignette: > %\VignetteIndexEntry{Sequential One-Way ANOVA} %\VignetteEncoding{UTF-8}{inputenc} %\VignetteEngine{knitr::rmarkdown} bibliography: references.bib csl: "apa.csl" --- ```{r, echo = FALSE, message = FALSE} knitr::opts_chunk$set( collapse = TRUE ) ``` The sequential one-way fixed effects ANOVA is a sequential hypothesis test based on the Sequential Probability Ratio Test (SPRT) framework [@wald1947]. It extends SPRTs to the comparison of two or more independent groups and can be used as an efficient alternative to the classical fixed-sample one-way ANOVA. For detailed information, see @steinhilber2024. For a general introduction to SPRTs, see the vignette `vignette("sprt")`. **Note:** The repeated measures ANOVA is not yet implemented in the `sprtt` package. ## Sequential One-Way ANOVA Analysis of variance (ANOVA) is widely used to compare means across multiple groups. Traditional fixed-sample ANOVAs require an a priori power analysis to determine the necessary sample size, which often results in large samples, especially when expected effect sizes are small [@steinhilber2024]. Given the prevalence of small to medium effects in psychology [@funder2019; @richard2003], many studies end up underpowered because the required sample sizes exceed available resources [@button2013; @szucs2017]. The sequential one-way ANOVA provides an efficient alternative by applying the SPRT framework to comparisons of $k$ groups. Instead of collecting a fixed number of observations, data are collected sequentially and the test evaluates evidence after each step until a decision is reached [@steinhilber2024]. ### Hypotheses The sequential one-way ANOVA tests the following hypotheses, specified in terms of Cohen's $f$: ```{=tex} \begin{aligned} H_0 &: f = 0 \\ H_1 &: f = f_{\text{exp}}, \quad (f_{\text{exp}} > 0) \end{aligned} ``` Cohen's $f$ is defined as: $$f = \frac{\sigma_m}{\sigma}$$ where $\sigma$ is the common within-population standard deviation and $\hat{\sigma}_m = \sqrt{\frac{\sum_{i=1}^{k}(m_i - \bar{m})^2}{k}}$, with $m_i$ being the mean of group $i$ and $\bar{m}$ the overall mean across all $k$ groups. In other words, $f$ captures the spread of group means relative to the common within-group variability. ### The $F$ Statistic At the $n$-th sequential step, the $F$ statistic is calculated from $k$ groups with $n$ observations each (total $N = k \cdot n$, with $n, k \geq 2$): $$F_n = \frac{SS_{\text{effect},n} / df_1}{SS_{\text{residual},n} / df_{2,n}}$$ with $$SS_{\text{effect},n} = \sum_{i=1}^{k} n(\bar{x}_i - \bar{x})^2$$ $$SS_{\text{residual},n} = \sum_{i=1}^{k} \sum_{j=1}^{n} (x_{i,j} - \bar{x}_i)^2$$ $$df_1 = k - 1 \qquad \text{and} \qquad df_{2,n} = N - k$$ where $\bar{x}$ is the overall mean, $\bar{x}_i$ is the mean of group $i$, and $x_{i,j}$ is the $j$-th observation in group $i$ [@wetherill1986; @steinhilber2024]. ### The Likelihood Ratio The likelihood ratio at step $n$ is defined as the ratio of the likelihood under $H_1$ to the likelihood under $H_0$. Using Cox's theorem [@cox1952], it is sufficient to compute this ratio only for the current $F_n$ statistic rather than the entire sequence of observations: $$\text{LR}_n = \frac{f(F_n \mid df_1,\; df_{2,n},\; \Delta_{1n})}{f(F_n \mid df_1,\; df_{2,n})}$$ The numerator is the density of a *non-central* $F$ distribution with non-centrality parameter $\Delta_{1n}$, and the denominator is the density of a *central* $F$ distribution (i.e., $\Delta = 0$ under $H_0$). The non-centrality parameter is linked to Cohen's $f$ via: $$\Delta_1 = f_{\text{exp}}^2 \cdot N$$ ### Decision Boundaries The sequential ANOVA uses two decision boundaries based on the specified error rates: - Upper boundary: $A = \frac{1-\beta}{\alpha}$ - Lower boundary: $B = \frac{\beta}{1-\alpha}$ where $\alpha$ is the Type I error rate and $\beta$ is the Type II error rate. ### Decision Rule At each analysis step, compare the likelihood ratio $\text{LR}_n$ to these boundaries: - If $\text{LR}_n \geq A$: Stop data collection and accept $H_1$ - If $\text{LR}_n \leq B$: Stop data collection and accept $H_0$ - If $B < \text{LR}_n < A$: Continue collecting data (no decision yet) ### Efficiency and Robustness Simulations with $k = 4$ groups demonstrate that the sequential ANOVA is substantially more efficient than fixed-sample designs: in 87% of cases the sequential sample was smaller than the fixed sample, with an average sample size reduction of 58%. Efficiency gains are particularly pronounced for small expected effect sizes [@steinhilber2024]. Robustness analyses showed that the sequential and fixed ANOVA behave similarly under assumption violations. Both procedures are robust to non-normal data (simulated using Gaussian mixture distributions) and to mildly unequal variances or group sizes. The most critical scenario is a combination of unbalanced group sizes and unequal variances, specifically when smaller groups have larger variances—in this case, $\alpha$ error rates can be inflated in both the sequential and the fixed ANOVA. As with all sequential procedures, effect size estimates from individual sequential ANOVAs are conditionally biased (see Section "The Bias-Efficiency Tradeoff"). However, the weighted average across studies closely approximates the true population effect size [@steinhilber2024]. The `seq_anova()` function in the `sprtt` package implements the sequential one-way fixed effects ANOVA described here. ## How to use `seq_anova()` ### Step 1: Simulate or Load Data First, we simulate data for this tutorial. In a real-world application, you would use actual data as it arrives from your ongoing data collection. ```{r example-0.1} set.seed(333) # Generate data with a medium effect data <- sprtt::draw_sample_normal( k = 3, # number of groups f = 0.25, # effect size (Cohen's f) max_n = 22 # maximum sample size per group ) ``` Let's examine the structure of the simulated data: ```{r example-0.1b} # View the first few rows head(data) # Check the sample sizes per group for the first 6 (2*k) data points table(data$x[1:6]) ``` The data frame contains two columns: `y` (the continuous outcome variable) and `x` (the grouping factor with 3 levels). ### Step 2: Initial Sequential Analysis We can perform the first sequential ANOVA after collecting at least 2 observations per group (minimum $n = 6$ total for 3 groups). ```{r example-0.2} # Calculate the sequential ANOVA anova_results <- sprtt::seq_anova( y ~ x, f = 0.25, data = data[1:6, ], verbose = FALSE ) # View results anova_results # Access the decision anova_results@decision ``` The decision indicates we should continue data collection. In practice, it's optimal to recalculate the sequential test after each new observation (or small batch of observations). ### Step 3: Repeated Sequential Testing As new data arrives, we *repeatedly recalculate* the sequential ANOVA. This is the core principle of sequential testing: check the decision criterion after each batch of new observations. Let's assume we've now collected 20 observations total. We recalculate the sequential ANOVA with the updated dataset: ```{r example-0.3} # Calculate sequential ANOVA with larger sample anova_results <- sprtt::seq_anova( y ~ x, f = 0.25, data = data[1:20, ], verbose = FALSE ) # View results anova_results # Check decision anova_results@decision ``` We still receive the decision to continue data collection. This means we repeat the process: collect more data, recalculate the sequential test, and check the decision again. This cycle continues until we reach a definitive decision to accept $H_0$ or $H_1$. ### Step 4: Final Analysis ```{r example-0.4} # Calculate sequential ANOVA with complete dataset anova_results <- sprtt::seq_anova( y ~ x, f = 0.25, data = data, verbose = TRUE ) # View full results anova_results ``` With a total sample size of $N =$ `r anova_results@total_sample_size`, we have reached a decision to `r anova_results@decision`. Therefore, we stop data collection. You can access specific components of the results object using the `@` operator: ```{r example-0.5} # Access the decision anova_results@decision # Access the likelihood ratio anova_results@likelihood_ratio # Access the total sample size anova_results@total_sample_size ``` ## How to plot the ANOVA results To visualize the likelihood ratio progression over time, `seq_anova()` must calculate the sequential test at each intermediate sample size. Since these calculations are only needed for plotting and increase computational time, they are performed only when `plot = TRUE`. ### Scenario 1: Balanced Data with Perfect Sampling Order When your data are perfectly balanced (equal $n$ per group) and sampled in order, you can set `plot = TRUE` and use either the default `seq_steps = "single"` or explicitly specify `seq_steps = "balanced"`. Both will produce accurate likelihood ratio trajectories. See `?seq_anova` for details on the `plot` and `seq_steps` arguments and other options. ```{r example-1} set.seed(333) data <- sprtt::draw_sample_normal(3, f = 0.25, max_n = 22) # calculate the SPRT ----------------------------------------------------------- # Default: plot = TRUE with seq_steps = "single" anova_results <- sprtt::seq_anova(y~x, f = 0.25, data = data, plot = TRUE) # Explicitly specify seq_steps = "single" anova_results <- sprtt::seq_anova(y~x, f = 0.25, data = data, plot = TRUE, seq_steps = "single") # Use balanced sequential steps anova_results <- sprtt::seq_anova(y~x, f = 0.25, data = data, plot = TRUE, seq_steps = "balanced") # plot the results ------------------------------------------------------------- sprtt::plot_anova(anova_results) ``` ### Scenario 2: Unbalanced Data with Imperfect Sampling Order In real-world applications, data often arrive with unequal group sizes and in imperfect group order. This scenario demonstrates how to handle such cases. **Why custom sequential steps are needed:** - The `"balanced"` option assumes equal $n$ across groups at each step, which doesn't match our data structure - The `"single"` option would produce an error because some groups have fewer than 2 observations in the initial first data points - **Solution:** Define custom sequential steps using a numeric vector In this example, we start the sequential testing after 12 observations (ensuring each group has sufficient data), then calculate the likelihood ratio after each subsequent observation. ```{r example-2} set.seed(333) # Generate unbalanced data with a 1:1:2 sampling ratio ------------------------- data <- sprtt::draw_sample_normal(3, f = 0.25, max_n = 37, sample_ratio = c(1,1,2)) # Randomize the order to get a more realistic data collection data <- data[sample(nrow(data)),] # Calculate the SPRT with custom sequential steps ------------------------------ anova_results <- sprtt::seq_anova( y~x, f = 0.25, data = data, plot = TRUE, # Start at n=12, then test after each observation seq_steps = 12:nrow(data)) # Plot the results with custom styling ----------------------------------------- sprtt::plot_anova(anova_results, labels = TRUE, position_labels_x = 0.2, position_labels_y = 0.2, position_lr_x = 100, position_lr_y = 1.8, font_size = 20, line_size = 1, highlight_color = "steelblue" ) ``` ## References