Fisherian and Neymanian Methods for Detecting and Measuring Treatment Effect Variation

Description

This package implements methods developed by Ding, Feller, and Miratrix (JRSS-B, 2016) "Randomization Inference for Treatment Effect Variation", for validly testing whether there is unexplained variation in treatment effects across observations. The package also implements methods introduced in Ding, Feller, and Miratrix (JASA, 2019) "Decomposing Treatment Effect Variation", for measuring the degree of treatment effect heterogeneity explained by covariates. The package includes wrapper functions implementing the proposed methods, as well as helper functions for analyzing and visualizing the results of the tests.

Details

This package partially supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305D150040. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Special thanks to Masha Bertling for some early work on documenting this project.

Author(s)

Peng Ding, Avi Feller, Ben Fifield, and Luke Miratrix

Maintainer: Ben Fifield benfifield@gmail.com

References

Ding, Peng, Avi Feller and Luke Miratrix. (2016) "Randomization Inference for Treatment Effect Variation", Journal of the Royal Statistical Society-Series B. Ding, Peng, Avi Feller and Luke Miratrix. (2019) "Decomposing Treatment Effect Variation", Journal of the American Statistical Association.

See Also

Useful links:

• Report bugs at https://github.com/bfifield/hettx/issues

KS_stat

Description

Calculate classic (not shifted) KS statistic; code is a modified version of R's ks.test().

Usage

KS_stat(Y, Z, tau = NULL, alternative = c("two.sided", "less", "greater"))

Arguments

Details

If tau passed, Y1 will be shifted by tau.

Value

The value of the test.

See Also

detect_idiosyncratic

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
KS_stat(df$Yobs, df$Z)

Sample data set

Description

This is a sample data set to illustrate the package methods.

Usage

Penn46_ascii

Format

A dataframe containing 6384 observations and 12 columns.

Estimate treatment variation R2

Description

Bounds the R2 measure (how much of treatment variation is explained by given covariates) using either the OLS output for the ITT from est.beta, or the LATE estimation from est.beta.

Usage

R2(est.beta, rho.step = 0.05)

Arguments

Value

RI.R2.result object.

See Also

print.RI.R2.result

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
es <- estimate_systematic( Yobs ~ Z,  interaction.formula = ~ A + B, data = df )
r2_out <- R2(es)

Extract the standard errors from a var-cov matrix.

Description

Extract the standard errors from a var-cov matrix.

Usage

SE(object, ...)

Arguments

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
es <- estimate_systematic( Yobs ~ Z,  interaction.formula = ~ A + B, data = df )
SE(es)

SKS_pool_t

Description

Subtract off group level treatment effect estimates and then look at KS statistic on residuals.

Usage

SKS_pool_t(Y, Z, W)

Arguments

Details

Distinct from the interacted lm in that the control units are not shifted and centered with respect to eachother.

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
df$W <- sample(c("A", "B", "C"), nrow(df), replace = TRUE)
SKS_pool_t(df$Yobs, df$Z, df$W)

SKS_stat

Description

Shifted kolmogorov-smirnov statistic. Calculate KS distance between Y0 and Y1 shifted by sample tau.

Usage

SKS_stat(Y, Z)

Arguments

Value

The value of the test.

See Also

KS_stat, SKS_stat_cov

detect_idiosyncratic

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
SKS_stat(df$Yobs, df$Z)

SKS_stat_cov_pool

Description

SKS_stat_cov_pool is the shifted kolmogorov-smirnov statistic with covariates to increase precision. This is the test statistic used Ding, Feller, and Miratrix (2016), JRSS-B.

SKS_stat_cov is the shifted kolmogorov-smirnov statistic with covariates with model for outcomes calculated on control group only. This avoids "splitting" the treatment variation between tx and co groups. We recommend this method over the "pool" method.

Usage

SKS_stat_cov_pool(Y, Z, X)

SKS_stat_cov(Y, Z, X)

Arguments

Value

The value of the test.

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
SKS_stat_cov_pool(df$Yobs, df$Z, df$A)

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
SKS_stat_cov(df$Yobs, df$Z, df$A)

SKS_stat_cov_rq

Description

Shifted kolmogorov-smirnov statistic with covariates and quantile regression.

Usage

SKS_stat_cov_rq(Y, Z, X)

Arguments

Value

The test statistic value.

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
SKS_stat_cov_rq(df$Yobs, df$Z, df$A)

SKS_stat_int_cov_pool

Description

SKS_stat_int_cov_pool is a shifted kolmogorov-smirnov statistic with a linear treatment effect model defined by W. It will attempt to remove any systematic variation corresponding to W and then return a SKS statistic on the residuals to measure any variation "left over".

SKS_stat_int_cov() is a Shifted kolmogorov-smirnov statistic with a linear treatment effect model defined by W. It will attempt to remove any systematic variation corresponding to W and then return a SKS statistic on the residuals to measure any variation "left over".

Usage

SKS_stat_int_cov_pool(Y, Z, W, X = NULL)

SKS_stat_int_cov(Y, Z, W, X = NULL)

Arguments

Details

X are _additional_ covariates to adjust for beyond those involved in treatment effect model. It will automatically ajust for W as well. Do not put a covariate in for both X and W.

This is the test statistic used in Ding, Feller, and Miratrix (2016), JRSS-B.

SKS_stat_int_cov first adjusts for baseline and then models treatment effect on the residuals to not split treatment effects (see the vignette for more information on this).

We recommend SKS_stat_int_cov over the "pool" method.

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
SKS_stat_int_cov_pool(Y = df$Yobs, Z = df$Z, W = df$A, X = df$B)


df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
SKS_stat_int_cov(Y = df$Yobs, Z = df$Z, W = df$A, X = df$B)

Toy data set

Description

This is a toy data set to illustrate the package methods.

Usage

ToyData

Format

A dataframe containing 500 observations and 7 columns.

WSKS_t

Description

Weighted average of the group-level SKS statistics. This is useful for a blocked experiment.

Usage

WSKS_t(Y, Z, W)

Arguments

Value

The value of the test.

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
df$W <- sample(c("A", "B", "C"), nrow(df), replace = TRUE)
WSKS_t(df$Yobs, df$Z, df$W)

Extract coefficients of a fit RI regression model.

Description

Extract coefficients of a fit RI regression model.

Usage

## S3 method for class 'RI.regression.result'
coef(object, ...)

Arguments

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
es <- estimate_systematic( Yobs ~ Z,  interaction.formula = ~ A + B, data = df )
coef(es)

detect_idiosyncratic

Description

Test for systematic treatment effect heterogeneity using Fisherian permutation inference methods.

Usage

detect_idiosyncratic(
  formula,
  data,
  interaction.formula = NULL,
  control.formula = NULL,
  plugin = FALSE,
  tau.hat = NULL,
  test.stat = ifelse(is.null(W) & is.null(X), "SKS_stat", ifelse(is.null(W),
    "SKS_stat_cov", "SKS_stat_int_cov")),
  te.vec = NULL,
  B = 500,
  gamma = 1e-04,
  grid.gamma = 100 * gamma,
  grid.size = 151,
  return.matrix = FALSE,
  na.rm = FALSE,
  n.cores = 1,
  verbose = TRUE,
  ...
)

Arguments

Value

If plug-in, the value of the test and the associated p-value. If not, a list with the value of the test statistic on the observed data, the value of the CI-adjusted p-value, the plug-in p-value, and other information on the test.

Examples

Z <- rep(c(0, 1), 100)
tau <- 4
Y <- ifelse(Z, rnorm(100, tau), rnorm(100, 0))
df <- data.frame(Y=Y, Z=Z)
tst <- detect_idiosyncratic(Y ~ Z, df, B = 50, grid.size = 50)

Calculate systematic effects model using LATE, ITT, or full potential outcomes.

Description

Implements the systematic effects model proposed in Ding, Feller, and Miratrix (2018). Can estimate an ITT or LATE model, or the actual beta in cases where full potential outcomes schedule is available.

Usage

estimate_systematic(
  formula,
  data,
  interaction.formula,
  control.formula = NULL,
  method = c("RI", "OLS", "2SLS"),
  na.rm = FALSE
)

Arguments

Details

The OLS method differs from the RI method only by how the Sxx matrix is handled. In the OLS case, seperate Sxx for treatment and control are calculated for each treatment arm. For RI the known Sxx based on all units is used.

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
es <- estimate_systematic( Yobs ~ Z,  interaction.formula = ~ A + B, data = df )

Generate permutation distributions for FRT tests

Description

Core engine used by FRTCI and FRTCI_interact to compute the randomization distribution of a test statistic under permuted treatment assignments.

Usage

generate_permutations(
  Y,
  Z,
  test.stat,
  Y0.mat,
  Y1.mat,
  B,
  n.cores,
  get.z.star = NULL,
  verbose = TRUE,
  ...
)

Arguments

Value

A list with components:

ks.obs

Observed test statistic.

ks.mat

B x grid.size matrix of permuted test statistics.

ci.p

Vector of p-values for each grid point.

get p-value along with uncertainty on p-value

Description

Give confidence bounds (from monte carlo simulation error) for the p-values returned by a test

Usage

get_p_value(tst)

Arguments

Value

p-value and range of p-values due to monte carlo error.

Examples

Z <- rep(c(0, 1), 100)
tau <- 4
Y <- ifelse(Z, rnorm(100, tau), rnorm(100, 0))
df <- data.frame(Y=Y, Z=Z)
tst <- detect_idiosyncratic(Y ~ Z, df, B = 50, grid.size = 50)
get_p_value( tst )

Generate a grid of treatment effect values to test

Description

Creates a sequence of tau values spanning a confidence interval around the estimated treatment effect, with oversampling near the point estimate.

Usage

get_tau_vector(
  Y,
  Z,
  X = NULL,
  gamma = 0.001,
  grid.size = 21,
  grid.gamma = 100 * gamma
)

Arguments

Value

A numeric vector of tau values with attributes te.hat, te.se, and te.MOE.

Build a grid of treatment effect models for interaction testing

Description

Estimates a linear model Y ~ Z + W + Z:W (+X) and samples from the confidence region of the Z-related coefficients. For each sampled model, computes individual imputed treatment effects and science tables of imputed potential outcomes.

Usage

get_testing_grid(Y, Z, W, X = NULL, gamma = 1e-04, grid.size = 150)

Arguments

Value

A list with components:

te.grid

grid.size x p matrix, each row a different effect model.

te.mat

N x grid.size matrix of individual treatment effects.

Y0.mat

N x grid.size matrix of imputed control potential outcomes.

Y1.mat

N x grid.size matrix of imputed treated potential outcomes.

Glance at a FRTCI.test result

Description

Glance at a FRTCI.test result

Usage

## S3 method for class 'FRTCI.test'
glance(x, ...)

Arguments

Value

A one-row data.frame with model-level summary statistics.

Glance at an RI.R2.result

Description

Glance at an RI.R2.result

Usage

## S3 method for class 'RI.R2.result'
glance(x, ...)

Arguments

Value

A one-row data.frame with model-level summary statistics.

Glance at an RI.regression.result

Description

Glance at an RI.regression.result

Usage

## S3 method for class 'RI.regression.result'
glance(x, ...)

Arguments

Value

A one-row data.frame with model-level summary statistics.

Deprecated functions in hettx

Description

These functions have been renamed to use snake_case. The old dot-case names still work but will emit a deprecation warning. Please update your code to use the new names.

Usage

test.stat.info()

KS.stat(...)

SKS.stat(...)

SKS.stat.cov.pool(...)

SKS.stat.cov(...)

SKS.stat.int.cov.pool(...)

SKS.stat.int.cov(...)

SKS.stat.cov.rq(...)

rq.stat(...)

rq.stat.cond.cov(...)

rq.stat.uncond.cov(...)

WSKS.t(...)

SKS.pool.t(...)

make.linear.data(...)

make.quadradic.data(...)

make.skew.data(...)

make.randomized.dat(...)

make.randomized.compliance.dat(...)

get.p.value(...)

variance.ratio.test(...)

Arguments

Generate dataset according to a linear model.

Description

Given the parameters, generate a dataset and return a potential outcomes schedule (science table) of synthetic potential outcomes.

Usage

make_linear_data(
  n,
  gamma.vec = c(1, 2, 2, 1),
  gamma2.vec = NULL,
  beta.vec = c(-1, -1, 1),
  ideo.sd = 0,
  quad.tx = FALSE,
  mu.X = FALSE,
  corr.X = TRUE
)

make_quadradic_data(n, beta.vec = c(-1, -1, 1))

make_skew_data(n, beta.vec = c(-1, -1, 1))

Arguments

Details

The control outcome surface is either linear or quadratic, of the form:

Y_i = \\gamma_0 + \\sum_{k=1}^J \\gamma_k X_{ki} + \\sum_{k=1}^{J_2} \\gamma^{(2)}_k X_{ki}^2 + \\epsilon_i

The individual treatment effects are similarly a linear or quadratic model.

Value

List of elements of data (not data frame)

Functions

• make_quadradic_data(): Generate dataset according to a quadratic model

• make_skew_data(): Generate dataset with a skew

Generate fake data with noncompliance.

Description

This will generate and randomize a science table to get observed outcomes and treatment assignment

Usage

make_randomized_compliance_dat(
  n,
  p = 0.6,
  science.table.generator = make_linear_data,
  include.POs = TRUE,
  ...
)

Arguments

Value

Data frame with data randomized to tx and control, and compliers, etc.

See Also

make_randomized_dat

Make fake data for simulations

Description

Randomize a science table to get observed outcomes and treatment assignment

Usage

make_randomized_dat(
  n,
  p = 0.6,
  science.table.generator = make_linear_data,
  include.POs = TRUE,
  as.data.frame = TRUE,
  ...
)

Arguments

Value

Either a list of elements or a dataframe.

plot.FRTCI.test

Description

Plot curve from FRTCI.test object.

Usage

## S3 method for class 'FRTCI.test'
plot(
  x,
  true.tau = NULL,
  xlab = expression(tau),
  ylab = "p-value",
  true.tau.col = "red",
  plot.envelope = TRUE,
  ci.line.col = "blue",
  ...
)

Arguments

Examples

Z <- rep(c(0, 1), 100)
tau <- 4
Y <- ifelse(Z, rnorm(100, tau), rnorm(100, 0))
df <- data.frame(Y=Y, Z=Z)
tst <- detect_idiosyncratic(Y ~ Z, df, B = 50, grid.size = 50)
plot(tst)

Make a plot of the treatment effect R2 estimates

Description

Make a plot of the treatment effect R2 estimates

Usage

## S3 method for class 'RI.R2.result'
plot(
  x,
  main = paste("R2 for Het Tx (", x$type, ")", sep = ""),
  ADD = FALSE,
  ...
)

Arguments

See Also

calc_beta_oracle

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
es <- estimate_systematic( Yobs ~ Z,  interaction.formula = ~ A + B, data = df )
r2_out <- R2(es)
plot(r2_out)

Generate truncated multivariate normal draws

Description

Draws from a multivariate normal distribution, rejecting any samples outside the confidence region defined by alpha.

Usage

rcrmtvnorm(n, mu, sigma, alpha)

Arguments

Value

An n x K matrix of draws from the truncated distribution.

rq_stat

Description

rq_stat is the Kolmogorov-smirnov statistic via quantile regression with covariates without further adjustment.

rq_stat_cond_cov does Kolmogorov-smirnov statistic via quantile regression with covariates, with a conditional approach; see Koenker and Xiao (2002).

rq_stat_uncond_cov implements a Kolmogorov-smirnov statistic via quantile regression with covariates, unconditional approach; see Firpo (2007).

Usage

rq_stat(Y, Z, rq.pts = seq(0.1, 0.9, by = 0.1))

rq_stat_cond_cov(Y, Z, X, rq.pts = seq(0.1, 0.9, by = 0.1))

rq_stat_uncond_cov(Y, Z, X, rq.pts = seq(0.1, 0.9, by = 0.1))

Arguments

Details

Warning: This function supresses all warnings of the 'rq()' method call.

Warning: This function supresses all warnings of the 'rq()' method call.

Value

The value of the test.

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
rq_stat(df$Yobs, df$Z)

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
rq_stat_cond_cov(df$Yobs, df$Z, df$A)

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
rq_stat_uncond_cov(df$Yobs, df$Z, df$A)

test_stat_info

Description

A list of test statistics for detect.idiosyncratic(), and information on use cases when each is appropriate.

Usage

test_stat_info()

Examples

test_stat_info()

Tidy a FRTCI.test result

Description

Tidy a FRTCI.test result

Usage

## S3 method for class 'FRTCI.test'
tidy(x, ...)

Arguments

Value

A data.frame with columns: statistic, p.value, p.value.plug, method, test.stat, estimate, std.error.

Tidy an RI.R2.result

Description

Tidy an RI.R2.result

Usage

## S3 method for class 'RI.R2.result'
tidy(x, ...)

Arguments

Value

A data.frame with R-squared bound estimates. For ITT results, one row. For LATE results, three rows (Compliers, Noncompliers, Covariates and compliers).

Tidy an RI.regression.result

Description

Tidy an RI.regression.result

Usage

## S3 method for class 'RI.regression.result'
tidy(x, ...)

Arguments

Value

A data.frame with one row per coefficient, containing columns: term, estimate, std.error.

Variance ratio test

Description

Given vector of observed outcomes and treatment vector, test to see if there is evidence the variances are different (taking kurtosis into account).

Usage

variance_ratio_test(Yobs, Z, data = NULL)

Arguments

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
variance_ratio_test(df$Yobs, df$Z)

Get vcov() from object.

Description

Get vcov() from object.

Usage

## S3 method for class 'RI.regression.result'
vcov(object, ...)

Arguments

Examples

df <- make_randomized_dat( 1000, gamma.vec=c(1,1,1,2), beta.vec=c(-1,-1,1,0) )
es <- estimate_systematic( Yobs ~ Z,  interaction.formula = ~ A + B, data = df )
vcov(es)