Confidence intervals and p-values

Every estimate ships with its uncertainty

A correlation coefficient on its own is hard to act on. smartcor therefore returns a full inference payload with every estimate: a confidence interval, a p-value, the null hypothesis being tested, and a short provenance string naming the source of each quantity. The guiding principle is that the method dictates the inference: a Kendall interval should not be computed the way a Pearson interval is, and the package picks the right machinery for you and tells you which it used.

The inference payload

Consider a routine continuous-continuous pair.

r = smart_cor(mtcars$mpg, mtcars$wt)
#> 
#> ── Variable types ──
#> 
#> ℹ Detected as continuous (numeric with 25 unique values).
#> mtcars$mpg: "continuous"
#> ℹ Detected as continuous (numeric with 29 unique values).
#> mtcars$wt: "continuous"
#> 
#> ── Method selection ──
#> 
#> ✔ Selected: Pearson Correlation

The console block already surfaces the interval, the p-value, the test, the null, and the source. For programmatic access, tidy() lays the same payload out as columns.

library(tibble)  # for printing
infcols = c("estimate", "statistic", "p.value", "p_method", "null_hypothesis",
             "ci_lower", "ci_upper", "conf_level", "ci_method", "ci_source")
as.data.frame(tidy(r)[infcols])
#>     estimate statistic      p.value               p_method null_hypothesis
#> 1 -0.8676594 -9.559044 1.293959e-10 t-test on r (cor.test)         rho = 0
#>     ci_lower   ci_upper conf_level ci_method           ci_source
#> 1 -0.9338264 -0.7440872       0.95  analytic Fisher z (cor.test)

Two fields are worth dwelling on. ci_method is either "analytic" (a closed-form or asymptotic interval) or "bootstrap" (a resampled interval), and ci_source names the exact reference or procedure behind it. Together they make the interval auditable: you can always see how it was produced, not just its endpoints.

Analytic intervals are the default

Thirteen of the fourteen methods carry a closed-form or asymptotic confidence interval, and smartcor uses it by default. Only Theil’s~U, which has no widely accepted analytic interval, falls back to resampling. The table below is generated by the package itself: for each method we force that estimator on a suitable pair and read back the inference provenance, so it cannot drift from what the code actually does.

set.seed(1)
colour = factor(sample(c("red", "blue", "green"), 90, replace = TRUE))
shape  = factor(sample(c("circle", "square", "triangle"), 90, replace = TRUE))

one = function(method, x, y) {
  tidy(smart_cor(x, y, method = method, verbose = FALSE))[
    c("method_label", "ci_method", "ci_source", "p_method", "null_hypothesis")]
}

spec = list(
  one("pearson",        mtcars$mpg,  mtcars$wt),
  one("spearman",       mtcars$mpg,  mtcars$wt),
  one("kendall",        mtcars$mpg,  mtcars$wt),
  one("point_biserial", mtcars$mpg,  mtcars$vs),
  one("phi",            mtcars$vs,   mtcars$am),
  one("tetrachoric",    mtcars$vs,   mtcars$am),
  one("yules_q",        mtcars$vs,   mtcars$am),
  one("polychoric",     mtcars$gear, mtcars$carb),
  one("polyserial",     mtcars$mpg,  mtcars$gear),
  one("gamma",          mtcars$gear, mtcars$carb),
  one("rank_biserial",  mtcars$vs,   mtcars$gear),
  one("cramers_v",      colour,      shape),
  one("theils_u",       colour,      shape),
  one("tschuprows_t",   colour,      shape)
)
map = do.call(rbind, spec)
knitr::kable(map[c("method_label", "ci_method", "ci_source")],
             row.names = FALSE, caption = "How each method's confidence interval is computed.")

The analytic sources are the standard ones: Fisher’s~\(z\) transform for Pearson and the point-biserial coefficient; Bonett and Wright (2000) for Spearman; Fieller, Hartley, and Pearson (1957) for Kendall; the asymptotic standard errors of Olsson (1979) and Olsson, Drasgow, and Dorans (1982) on the Fisher-\(z\) scale for the latent-variable methods (polychoric, tetrachoric, polyserial); Brown and Benedetti (1977) for Goodman-Kruskal’s gamma and Yule’s~Q; Cliff (1996) for the rank-biserial coefficient; and a noncentral chi-square pivot for the chi-square-based association measures (phi, Cramer’s~V, Tschuprow’s~T). The noncentral pivot deserves a note: it inverts the chi-square test for the noncentrality parameter, so its intervals are asymmetric and correctly bound at zero when the association is weak, rather than spilling below it. One caveat: the point-biserial interval reuses the Fisher-\(z\) variance \(1/(n-3)\), which assumes bivariate normality; a binary margin cannot satisfy that exactly, so treat this interval as an approximation.

The bootstrap argument

The bootstrap argument is a tri-state control.

• "auto" (the default) uses the analytic interval when one exists and is finite, and resamples only when it does not, for example when a latent-variable optimiser returns a singular Hessian or a contingency table is too sparse for the pivot.
• TRUE always adds a percentile bootstrap, which is useful for cross-checking an analytic interval against a distribution-free one.
• FALSE reports only the analytic interval, returning NA endpoints if none is available.

The number of resamples is set by n_boot (default 500). Here we ask for a bootstrap interval on the same Pearson pair and compare it against the Fisher-\(z\) interval above.

rb = smart_cor(mtcars$mpg, mtcars$wt, bootstrap = TRUE, n_boot = 1000, verbose = FALSE)
tidy(rb)[c("estimate", "ci_lower", "ci_upper", "ci_method", "ci_source")]
#> # A tibble: 1 × 5
#>   estimate ci_lower ci_upper ci_method ci_source           
#>      <dbl>    <dbl>    <dbl> <chr>     <chr>               
#> 1   -0.868   -0.925   -0.792 bootstrap Percentile bootstrap

The two intervals agree closely here, so the asymptotic approximation holds up on this pair. Pass seed if you need the resampled interval to be reproducible; a seeded call restores your .Random.seed on exit. The confidence level is controlled by conf_level; lowering it narrows the interval.

rbind(
  `90%` = tidy(smart_cor(mtcars$mpg, mtcars$wt, conf_level = 0.90, verbose = FALSE))[c("ci_lower", "ci_upper")],
  `95%` = tidy(smart_cor(mtcars$mpg, mtcars$wt, conf_level = 0.95, verbose = FALSE))[c("ci_lower", "ci_upper")]
)
#> # A tibble: 2 × 2
#>   ci_lower ci_upper
#> *    <dbl>    <dbl>
#> 1   -0.926   -0.769
#> 2   -0.934   -0.744

The null hypothesis is method-specific

A correlation p-value is only meaningful against a stated null, and the natural null differs across methods. Pearson, Spearman, Kendall, and the latent-variable methods test whether the (latent) correlation is zero; the chi-square-based measures test independence of the two variables; Theil’s~U tests whether one variable carries no information about the other. The null_hypothesis and p_method columns make this explicit, so a small p-value is never ambiguous about what has been rejected.

knitr::kable(unique(map[c("method_label", "p_method", "null_hypothesis")]),
             row.names = FALSE, caption = "The null hypothesis and test behind each p-value.")

Reading interval widths

Interval widths are worth reading, not just the point estimates. compare_methods() reports every applicable method for a pair side by side, with each interval on its own terms.

compare_methods(mtcars$gear, mtcars$carb)
#> 
#> ── Comparing methods ──
#> 
#> mtcars$gear ("ordinal") × mtcars$carb ("ordinal")
#> N: 32
#> Methods: "polychoric", "kendall", "spearman", and "gamma"
#> 
#> "polychoric": 0.2451 ← recommended
#> "kendall": 0.0980
#> "spearman": 0.1149
#> "gamma": 0.1405
#> 
#> ── Method Comparison ───────────────────────────────────────────────────────────
#> Variables: mtcars$gear ("ordinal") × mtcars$carb ("ordinal")
#> N: 32
#> 
#> Polychoric Correlation ← recommended
#> r = 0.2451 p = 0.2490 95% CI [-0.1734, 0.5886]
#> Kendall's Tau-b
#> r = 0.0980 p = 0.5302 95% CI [-0.1455, 0.3303]
#> Spearman Rank Correlation
#> r = 0.1149 p = 0.5312 95% CI [-0.2447, 0.4467]
#> Goodman-Kruskal's Gamma
#> r = 0.1405 p = 0.5433 95% CI [-0.3046, 0.5353]

The Goodman-Kruskal gamma interval is much wider than the rank-based ones. Gamma discards all tied pairs, so its effective sample size is smaller and its interval is correspondingly less precise. Two methods with similar coefficients can differ a lot in how firmly those coefficients are pinned down.

Summary

Every result carries an interval, a p-value, an explicit null, and a named source. Analytic intervals are the default for all but one method, and the bootstrap argument covers the rest: a distribution-free cross-check when you want one, and a fallback when an analytic interval is unavailable.