Psychology | Methods
Sequential t-Tests
Introduction
sprt.t.test is a function that implements Abraham Wald’s (1947) Sequential Probability Ratio Test (SPRT) for the test of a normal mean (difference) with unknown variance in R (R Core Team, 2020). It performs sequential t tests developed by Rushton (1950, 1952) and Hajnal (1961), based on the SPRT. Specifically, sprt.t.test performs
- one-sample, two-sample, and paired t-tests for testing
- one- and two-sided hypotheses.
To use sprt.t.test, please install the R software environment. The test is to be applied to the data during the sampling process, ideally after each observation. At any stage, it will return a decision to either continue sampling or terminate and accept one of the specified hypotheses. For more information on the SPRT t test, see Schnuerch and Erdfelder (2020).
Set up sprt.t.test
To set up the function, you need to download, unzip, and execute the script file Sequential t tests.R. The script contains five help functions and the main function. When executing the script, the functions are defined and made available in the global environment. The help functions need not be called directly to perform the test, you only need to call the main function sprt.t.test.
To execute the script, you can open the script or copy the code into your analysis script and run it. Alternatively, you can use the following command in your analysis script (make sure that the file is in your current working directory or specify the path):
source("Sequential t tests.R")Input
Function arguments
The function is based on the t.test function in R’s {stats} package. Thus, its argument structure resembles that of t.test:
## Default S3 method:
sprt.t.test(x, y = NULL,
mu = 0, d, alpha = 0.05, power = 0.95,
alternative = "two.sided", paired = FALSE)
## S3 method for class 'formula'
sprt.t.test(formula, data = NULL,
mu = 0, d, alpha = 0.05, power = 0.95,
alternative = "two.sided", paired = FALSE)| Arguments | |
|---|---|
| x, y | non-empty numeric vectors containing the data values. Note that y is optional and only required for two-sample tests (see subsection Data). |
| formula | a formula of the form lhs ~ rhs, with lhs denoting a (non-empty) numeric vector of data values and rhs a factor with exactly two levels representing the corresponding groups (see subsection Data). |
| mu | a number indicating the true value of the mean (or mean difference) under the null hypothesis. The default is 0. |
| d | a number indicating the expected standardized effect size (Cohen’s d in the population). No default value is given, must be specified by the user. |
| alpha, power | Type I error probability and power (1 – Type II error probability) of the test. Default values are 0.05 and 0.95, respectively. |
| alternative | a character string indicating whether the alternative hypothesis is one- or two-sided. Must be one of ‘two.sided’ (default), ‘greater’, or ‘less’. |
| paired | a logical indicating whether the test is based on dependent (paired, matched) samples. The default is FALSE. This argument is ignored in one-sample tests. |
Data
Like t.test, sprt.t.test accepts flexible data input. Depending on the data argument, the function will perform different tests. For a one-sample test, the function requires a single (non-empty) numeric vector. If the vector is not numeric, the function returns an error. If a second (non-empty) numeric vector is specified, it will perform a (paired or independent) two-sample test. Each of the two vectors is expected to contain data from one of the two samples, respectively. Thus, the function will return an error if one of the vectors is not numeric.
sprt.t.test(x, y = NULL, d = 0.5) # one sample
sprt.t.test(x, y, d = 0.5) # two independent samples
sprt.t.test(x, y, , d = 0.5, paired = TRUE) # two paired samples## if the data are not numeric
x.example <- rnorm(30)
y.example <- as.logical(rnorm(30))
sprt.t.test(x.example, y.example, d = 0.5)## Error in .sprt.default(x.example, y.example, d = 0.5): Invalid argument: y.example must be numeric.A two-sample t test can also be performed using the formula interface. The formula must be of the form lhs ~ rhs, where lhs is a numeric vector which contains the data values and rhs is a factor containing exactly two levels representing the corresponding groups. If rhs is not a factor, it will be coerced automatically.
sprt.t.test(x ~ y, data = NULL, d = 0.5) # two independent samples
sprt.t.test(x ~ y, data = NULL, d = 0.5, paired = TRUE) # two paired samplesdata is an optional argument specifying a matrix or data.frame containing the variables in the formula. If no data argument is specified, the variables will be searched in the global environment. Note that a paired t test requires both groups to be of equal length and in the same order (as the test is based on pairwise differences). We do not recommend using the formula interface in this case but sprt.t.test will accept paired = TRUE when a formula is specified.
If the data contain missing values, these will be silently removed (listwise). If the data are effectively constant, that is, within-groups variance estimates are 0, the function will return an error.
The alternative argument “greater” (and, correspondingly, “less”) represents the one-sided hypothesis that
- mean(x) > mu
- mean(x) - mean(y) > mu
- mean(x in group 1) - mean(x in group 2) > mu
When using the formula interface, group 1 and 2 are defined by the levels of the factor. Unless explicitly defined otherwise, level ordering will be in increasing (e.g., alphabetical) order. Keep that in mind when specifying the alternative argument (as “greater” or “less”).
Output
sprt.t.test returns a list of class “sprt” containing the following components:
| Component | |
|---|---|
| statistic | the value of the likelihood-ratio statistic: P(data;H1) / P(data;H0) |
| decision | the decision of the SPRT at this stage of the sampling process; one of ‘accept H1’ (when LR >= A), ‘accept H0’ (when LR <= B), or ‘continue sampling’ (when B < LR < A). |
| parameters | the error probabilities (or power) of the test as specified by the user. |
| thresholds | the likelihood-ratio threshold values B (lower) and A (upper) for the sequential procedure (based on the specified error probabilities). |
| estimate | the estimated mean (or difference in means) of the data. |
| null.value | the specified value of the mean (or difference in means) under the null hypothesis. |
| alternative | a character string denoting the alternative hypothesis as specified by the user. |
| effect.size | the specified expected effect size under the alternative hypothesis (Cohen’s d in the population). |
| method | a character string indicating what type of t-test was performed (one-sample, two-sample, or paired). |
| data.name | a character string containing the name(s) of the data vector(s). |
Sequential t tests.R contains the function print.sprt, which defines a generic print function for objects of class “sprt”. Thus, when printing an object returned by sprt.t.test R will display it in a customized way, resembling the t.test output.
## example 1: one-sample t-test
x <- rnorm(50)
sprt.t.test(x, d = 0.5)## One-Sample SPRT t-test
##
## data: x
## likelihood ratio = 0.00969, decision = accept H0
## SPRT thresholds:
## lower upper
## 0.05263 19.00000
## alternative hypothesis: true mean is not equal to 0
## effect size: Cohen's d = 0.5
## Type I error Power
## 0.05 0.95
## sample estimates:
## mean of x
## -0.09169## example 2: two-sample t-test
x <- rnorm(50)
y <- as.factor(sample(c(1,2), 50, replace = TRUE))
x <- ifelse(y == 1, x + 0.8, x)
sprt.t.test(x ~ y, d = 0.8, alternative = "greater")## Two-Sample SPRT t-test
##
## data: x by y
## likelihood ratio = 404.5057, decision = accept H1
## SPRT thresholds:
## lower upper
## 0.05263 19.00000
## alternative hypothesis: true difference in means is greater than 0
## effect size: Cohen's d = 0.8
## Type I error Power
## 0.05 0.95
## sample estimates:
## mean in group 1 mean in group 2
## 0.90545 -0.11290## example 3: paired t-test
x <- rnorm(20)
y <- rnorm(20)
sprt.t.test(x, y, d = 0.5, paired = TRUE, alternative = "less")## Paired SPRT t-test
##
## data: x and y
## likelihood ratio = 0.06178, decision = continue sampling
## SPRT thresholds:
## lower upper
## 0.05263 19.00000
## alternative hypothesis: true difference in means is less than 0
## effect size: Cohen's d = 0.5
## Type I error Power
## 0.05 0.95
## sample estimates:
## mean of the differences
## 0.04203Concluding Remarks
Sequential t tests.R and sprt.t.test may be used for non-commercial purposes free of charge. Although considerable effort was put into developing and testing the functions provided herein, there is no warranty whatsoever. Please refer to Schnuerch and Erdfelder (2020) for further information on the SPRT and sequential t tests. We are grateful for comments, questions, or suggestions. Please address communication concerning sprt.t.test to Martin Schnuerch.