The Pearson product-moment correlation coefficient is the most widely used measure of linear association. However, it is only optimal when both variables are continuous and approximately normally distributed. In practice, researchers work with binary variables (yes/no, male/female), ordinal variables (Likert scales, education levels), and nominal categorical variables (colour, country), each requiring a different correlation method to avoid bias, attenuation, or outright meaninglessness.
smartcor automates method selection based on detected variable types. This vignette explains the statistical theory behind each method, why it is appropriate for a given variable-type combination, and what can go wrong when the wrong method is used.
Before selecting a correlation method, we must classify each variable. smartcor recognises five types:
| Type | Description | Examples |
|---|---|---|
| Continuous | Numeric with many distinct values | Income, temperature, test scores |
| Count | Non-negative integers with many distinct values | Number of citations, visits, or events |
| Binary | Exactly two distinct values | Male/female, yes/no, 0/1 |
| Ordinal | Ordered categories, discrete | Likert scales (1–5), education level |
| Categorical | Unordered categories | Colour, country, blood type |
The detect_type() function applies the following
heuristic, in order:
ordinal_threshold).detect_count = FALSE to skip
this step).This heuristic is deliberately conservative: a 7-point Likert scale stored as integers (1, 2, …, 7) will be detected as ordinal, which is the correct statistical treatment. Count variables are treated exactly like continuous variables when a correlation is computed; the separate label is purely descriptive.
The Pearson correlation coefficient between continuous variables \(X\) and \(Y\) is defined as:
\[r = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2 \cdot \sum_{i=1}^n (y_i - \bar{y})^2}}\]
Why Pearson? For two continuous variables, Pearson is the maximum-likelihood estimator of the population correlation \(\rho\) under bivariate normality. It captures linear association and has well-understood sampling properties (exact \(t\)-test for \(H_0: \rho = 0\), Fisher’s \(z\)-transformation for confidence intervals).
When Pearson is suboptimal: Pearson is sensitive to
outliers and non-linear relationships. When the relationship is
monotonic but not linear, or when distributions are heavily skewed,
Spearman or Kendall may be more appropriate. smartcor offers these as
alternatives via method = "spearman" or
method = "kendall".
When one variable is continuous and the other is binary (coded 0/1), the point-biserial correlation is used. It is defined as:
\[r_{pb} = \frac{\bar{X}_1 - \bar{X}_0}{s_X} \cdot \sqrt{\frac{n_1 n_0}{n^2}}\]
where \(\bar{X}_1\) and \(\bar{X}_0\) are the means of the continuous variable in each binary group, \(s_X\) is the pooled standard deviation, and \(n_1\), \(n_0\) are the group sizes.
Key insight: The point-biserial correlation is mathematically identical to the Pearson correlation when the binary variable is coded as 0 and 1. This is not an approximation; it is an algebraic equivalence. The proof follows from substituting \(Y \in \{0, 1\}\) into the Pearson formula and simplifying (see Appendix in Harshvardhan and Ranjan, 2026).
Monte Carlo simulations confirm this: the maximum absolute difference between Pearson and point-biserial across 500 replications, multiple sample sizes, and multiple true correlations is exactly 0.
Implication: There is no advantage to using a specialised point-biserial function in software. The Pearson correlation with 0/1 coding gives the identical result. smartcor uses this equivalence internally.
When both variables are binary, the phi coefficient (\(\phi\)) measures their association. For a \(2 \times 2\) contingency table:
\[\phi = \frac{ad - bc}{\sqrt{(a+b)(c+d)(a+c)(b+d)}}\]
where \(a, b, c, d\) are the cell counts.
Phi = Pearson for 0/1 coding. Like the point-biserial case, \(\phi\) is algebraically identical to the Pearson correlation when both variables are coded 0/1. Simulations confirm that the maximum difference is on the order of \(10^{-14}\) (floating-point precision).
The attenuation problem. Phi measures the observed association between two binary variables. But what if each binary variable is a dichotomised version of an underlying continuous (latent) variable? For example, “pass/fail” is a binary reduction of a continuous test score.
In this case, phi underestimates the latent correlation \(\rho\). The attenuation depends on the marginal proportions (how far the cut points are from the median):
This is why tetrachoric correlation exists.
The tetrachoric correlation assumes that each observed binary variable arises from dichotomising an underlying continuous normal variable at some threshold. It estimates the correlation between the latent continuous variables, not the observed binary ones.
The model is:
\[(X^*, Y^*) \sim \text{Bivariate Normal}(\mu, \Sigma) \quad \text{with} \quad \Sigma = \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}\]
The observed binary variables are \(X = \mathbf{1}(X^* > \tau_x)\) and \(Y = \mathbf{1}(Y^* > \tau_y)\), where \(\tau_x, \tau_y\) are the thresholds estimated from the marginal proportions.
When to use tetrachoric:
When NOT to use tetrachoric:
A 2 x 2 table is always saturated (df = 0), so the LR test that
drives assume_latent_normal = "auto" cannot run and the
data cannot support the latent-normality assumption. smartcor therefore
defaults a binary-binary pair to phi; tetrachoric requires an explicit
assume_latent_normal = TRUE or
method = "tetrachoric".
The polychoric correlation generalises the tetrachoric to ordinal variables with more than two categories. It assumes each observed ordinal variable is a discretisation of a latent normal variable:
\[X^* \sim N(0, 1), \quad X = k \iff \tau_{k-1} < X^* \leq \tau_k\]
The polychoric correlation \(\hat{\rho}\) is the maximum-likelihood estimate of \(\rho\) in the bivariate normal model, given the observed contingency table and estimated thresholds.
Advantages:
Disadvantages:
Alternative: Kendall’s tau-b (\(\tau_b\)).
Kendall’s tau is defined in terms of concordant and discordant pairs:
\[\tau_b = \frac{C - D}{\sqrt{(C + D + T_X)(C + D + T_Y)}}\]
where \(C\) = concordant pairs, \(D\) = discordant pairs, \(T_X\) = pairs tied on \(X\) only, \(T_Y\) = pairs tied on \(Y\) only.
Simulations show that Kendall’s tau has a consistent negative bias relative to the true \(\rho\) (it underestimates), but this bias is stable across all distribution types: normal, skewed, heavy-tailed, bimodal, and uniform. This makes it the most robust choice when the latent normality assumption is uncertain.
smartcor defaults to polychoric when latent normality is assumed, and Kendall’s tau otherwise.
The polyserial correlation is the continuous-ordinal analogue of polychoric. It assumes the ordinal variable arises from discretising a latent normal variable, while the continuous variable is observed directly.
The model estimates \(\rho\) between the continuous variable and the latent continuous variable underlying the ordinal one.
Simulations show polyserial is essentially unbiased (bias < 0.005) when the latent normality assumption holds. When the assumption is violated, Spearman’s rank correlation is a robust alternative.
When one variable is binary and the other is ordinal, the rank-biserial correlation (Glass, 1966) is appropriate. It compares the mean ranks of the ordinal variable in each binary group:
\[r_{rb} = \frac{2(\bar{R}_1 - \bar{R}_0)}{n}\]
This is a nonparametric measure that respects the ordinal nature of one variable and the dichotomous nature of the other. Kendall’s tau is available as an alternative.
When both variables are nominal (unordered categories), there is no concept of “direction” in the association. Cramer’s V is the standard symmetric measure:
\[V = \sqrt{\frac{\chi^2}{n \cdot (k - 1)}}\]
where \(\chi^2\) is the Pearson chi-squared statistic and \(k = \min(r, c)\) is the smaller of the number of rows and columns in the contingency table.
Cramer’s V ranges from 0 (no association) to 1 (perfect association) and is corrected for table dimensions.
Alternatives:
A common question is: “Why not just use Pearson for everything?” The answer depends on what you mean by correlation and what you intend to do with it.
1. Attenuation bias. When continuous variables are observed only as ordinal categories (e.g., Likert scales), Pearson/phi/point-biserial measures the association between the observed (discretised) values. This systematically underestimates the latent association. For example, with a true \(\rho = 0.7\) and binary dichotomisation, phi gives approximately 0.45–0.50. Polychoric/tetrachoric recovers the correct 0.70.
This matters for factor analysis and structural equation modelling, where underestimated correlations lead to underestimated factor loadings and distorted model fit.
2. Scale appropriateness. Pearson assumes interval-scale data. Ordinal variables violate this: the “distance” between categories 1 and 2 is not necessarily the same as between 4 and 5. Rank-based measures (Kendall, Spearman) and latent-variable measures (polychoric) respect the ordinal scale.
3. Meaninglessness for nominal data. Computing Pearson on arbitrarily coded nominal variables (e.g., red = 1, blue = 2, green = 3) is meaningless because the numbers have no inherent order. Cramer’s V and related measures correctly capture nominal association.
The choice between latent-variable methods (polychoric, polyserial, tetrachoric) and distribution-free methods (Kendall, Spearman) hinges on whether the underlying latent distribution is approximately normal.
When latent normality is reasonable:
When latent normality is doubtful:
Practical guidance:
The following table summarises smartcor’s decision matrix:
| Variable types | Default method | Alternative |
|---|---|---|
| Correlation methods (signed, from -1 to +1) | ||
| Continuous + continuous | Pearson | Spearman (suggested, especially if nonlinear) |
| Continuous + binary | Pearson = point-biserial | — |
| Binary + binary | Pearson = phi | Tetrachoric (preferred, if latent normality holds) |
| Continuous + ordinal | Spearman/Kendall | Polyserial (preferred, if latent normality holds) |
| Ordinal + ordinal | Kendall’s tau | Polychoric (preferred, if latent normality holds) |
| Binary + ordinal | Rank-biserial | Spearman |
| Association measures (unsigned, from 0 to 1) | ||
| Continuous + categorical | Cramer’s V (binned) | — |
| Binary + categorical | Cramer’s V | — |
| Ordinal + categorical | Cramer’s V | — |
| Categorical + categorical | Cramer’s V | Theil’s U, Tschuprow’s T |
Count variables are treated as numeric continuous variables, so each count combination follows the corresponding continuous row.
By default (assume_latent_normal = "auto"), smartcor
runs the likelihood-ratio chi-square test of
test_bivariate_normality() on each pair where the
latent-normality question matters and follows the verdict: polychoric or
polyserial when the bivariate normal threshold model is not rejected,
Kendall or Spearman when it is. The binary-binary pair is decided by the
saturated-table rule above: the 2 x 2 table gives the test zero degrees
of freedom, so phi is reported unless
assume_latent_normal = TRUE is set. Passing
NULL asks interactively; in non-interactive sessions that
falls back to TRUE.
The smart_cor_df() convenience function uses
assume_latent_normal = "auto" and never prompts, which
makes it suitable for scripts and pipelines.