An integrated workflow for factorial survey (vignette) experiments in R.
A factorial survey shows respondents short descriptions whose
attributes are experimentally varied, and asks for a rating – typically
on a bounded scale such as 0 to 10 – or a yes/no answer. Because the
attributes are randomised by design, their effects on the rating carry a
causal interpretation. FactorialSurvey covers the stages of
such a study in one place: building an efficient set of vignettes,
checking how many respondents are needed, estimating a model suited to
bounded ratings or yes/no answers with standard errors clustered by
respondent, and translating the estimates into quantities that can be
reported: marginal willingness to pay for each attribute and expected
willingness to pay for complete policy scenarios.
See vignette("FactorialSurvey") for a walk-through with
real data.
# install.packages("remotes")
remotes::install_github("tanakak/FactorialSurvey")fs_defficient() picks a D-efficient subset of the full
factorial and splits it into blocks, one per respondent group.
library(FactorialSurvey)
attributes <- list(
damage = c("minor", "moderate", "severe"),
distance = c("near", "far"),
cost = c(100, 500, 1000, 3000)
)
design <- fs_defficient(
attributes,
n_vignettes = 24,
n_blocks = 4,
seed = 2026,
verbose = FALSE
)
designCombinations that would confuse a respondent can be ruled out before the search:
design <- fs_defficient(
attributes,
n_vignettes = 24,
n_blocks = 4,
restrictions = ~ damage == "minor" & cost == 3000,
seed = 2026,
verbose = FALSE
)fs_design_eval() reports the D- and A-efficiency (100
percent for an orthogonal array), how evenly the levels appear and how
strongly the attributes are correlated. fs_efficiency()
gives the efficiency of any design, including one built elsewhere.
fs_design_eval(design)Before fielding, fs_power() simulates the survey under
effects you expect and reports how often each would be detected, and how
precise willingness to pay would be, for several sample sizes.
set_labels() attaches the wording the respondent will
actually see:
design <- set_labels(
as.data.frame(design),
list(damage = c("Minor", "Moderate", "Severe"),
distance = c("Nearby", "Far away"))
)fs_vignette_text() fills a template with each row’s
levels. Formatting functions render numbers the way respondents should
read them, without altering the design columns.
design <- fs_vignette_text(
design,
"A {{damage}} damaged building {{distance}}. Repairs cost {{cost}} a year.",
format = list(cost = function(x) paste0("$", formatC(x, big.mark = ",")))
)fs_assign() deals blocks out to respondents and orders
the vignettes within each block using a Williams design, so that no
attribute is confounded with position or with the vignette a respondent
happened to see just before.
schedule <- fs_assign(design, n_respondents = 200, seed = 1)fs_export() writes the result out. CSV files carry a
UTF-8 byte order mark by default, which is what makes non-ASCII text
open correctly in Excel on Windows.
fs_export(design, "vignettes.csv")
fs_export(design, "loop.csv", format = "qualtrics") # one file per block
fs_export(schedule, "schedule.xlsx", format = "xlsx") # one sheet per blockfs_betareg() maps the rating scale onto the open unit
interval and fits a beta regression, which respects the bounds of the
scale instead of assuming an unbounded normal error.
fit <- fs_betareg(rating ~ damage + distance + cost,
data = survey_data,
scale = c(0, 10),
cluster = ~ respondent_id)
summary(fit)Because each respondent rates several vignettes, the ratings of one
person are not independent. cluster makes standard errors,
marginal effects and willingness-to-pay intervals robust to this;
without it fs_betareg() warns.
Average marginal effects are reported both on the unit interval and rescaled back onto the original 0–10 scale, so they can be read as “this attribute moves the rating by so many points”.
w <- fs_mwtp(fit, price = "cost")
plot(w)
fs_mwtp(fit, price = "cost", method = "krinsky-robb", seed = 1)
# Expected willingness to pay for complete scenarios
scenarios <- data.frame(damage = c("minor", "severe"),
distance = c("far", "near"),
row.names = c("mild", "serious"))
fs_ewtp(fit, scenarios, price = "cost")MWTP is a ratio of coefficients, so its sampling distribution is skewed. Three interval methods are offered – the delta method, Krinsky-Robb simulation and Fieller’s theorem – and it is worth comparing them before reporting a result.
| Stage | Functions |
|---|---|
| Design | fs_defficient(), fs_efficiency(),
fs_design_eval() |
| Planning | fs_simulate_ratings(), fs_power() |
| Fielding | set_labels(), fs_vignette_text(),
fs_williams(), fs_assign(),
fs_export() |
| Estimation | fs_betareg() (ratings), fs_logit()
(yes/no) |
| Willingness to pay | fs_mwtp(), fs_ewtp() |
| Graphics | plot() methods, theme_fs() |
| Data | bento |
Planned for future releases:
FactorialSurvey is about rating-based vignette
experiments. For choice-based designs and analyses see idefix
and cregg.
The efficient design search here is built on AlgDesign
and the estimation on betareg.
MIT