robust_yeojohnson: Robust Yeo-Johnson Transformation

View source: R/robust_yeojohnson.R

robust_yeojohnsonR Documentation

Robust Yeo-Johnson Transformation

Description

Transforms each variable toward central normality with the Yeo-Johnson transformation, whose parameter \lambda is fitted robustly (Raymaekers and Rousseeuw, 2021), so that outlying observations do not drive it. Unlike the Box-Cox transformation (robust_boxcox()), it accepts zero and negative values. robust = FALSE gives the classical fit, by maximum likelihood, for comparison.

Usage

robust_yeojohnson(
  x,
  quantile = 0.99,
  nbsteps = 2,
  standardize = TRUE,
  robust = TRUE
)

Arguments

x

A numeric vector, matrix, data frame or tibble, with one variable per column (such as line intensities, contents or spectral channels).

quantile

The quantile of the absolute value of a standard normal variable beyond which the standardized transformed values are given weight zero in the re-weighting steps, strictly between 0 and 1. Default is 0.99.

nbsteps

The number of re-weighting steps, a positive integer. Default is 2.

standardize

A logical value: standardize the transformed variables by the mean and standard deviation of the inliers (TRUE, default).

robust

A logical value: fit \lambda robustly (TRUE, default), or by classical maximum likelihood on all the values (FALSE), to compare (see Robust fit).

Details

The Yeo-Johnson transformation (Yeo and Johnson, 2000) of a value x \ge 0 is

\psi_\lambda(x) = \frac{(1 + x)^\lambda - 1}{\lambda}

(\log(1 + x) for \lambda = 0), and of a value x < 0

\psi_\lambda(x) = -\frac{(1 - x)^{2 - \lambda} - 1}{2 - \lambda}

(-\log(1 - x) for \lambda = 2). \lambda = 1 leaves the values unchanged; a \lambda below 1 compresses the upper tail, for right-skewed variables, and a \lambda above 1 compresses the lower tail, for left-skewed ones.

Each variable is first centered by its median and divided by its MAD, so that \lambda does not depend on its location or units. It describes the shape of the distribution on this robust standard scale, and cannot be compared with the \lambda of robust_boxcox(): on the same data, it is often much closer to 1.

It suits the variables with zero or negative values, such as baseline-corrected, derivative or SNV-scaled spectra. Strictly positive variables, such as raw intensities or contents, can be transformed with either transformation.

Value

An object of class specproc_yeojohnson (a list), which predict() applies to new data, or with inverse = TRUE uses to transform values back to the original scale (as a median or a mean), with the following components:

  • transformation: A tibble of the transformed variables.

  • summary: A tibble with one row per variable: variable, lambda (NA for the variables left unchanged) and objective (the robust distance between the transformed values and the normal quantiles of step 2; lower is closer to normality).

  • fits: The fitted parameters of each variable, used by predict().

  • standardize: The standardize argument.

  • robust: The robust argument.

Robust fit

The parameter \lambda of each variable is fitted as follows:

  1. The variable is pre-standardized (see Details).

  2. An initial \lambda (between -4 and 6) minimizes a robust distance (Tukey's biweight) between the sorted, robustly standardized transformed values and the quantiles of the normal distribution. For this step, the transformation is continued linearly on the side that it compresses (the upper side for \lambda < 1, the lower side for \lambda > 1), beyond the point where the transformed value is 1.5 times that of the quartile, so that outliers on that side cannot dictate \lambda.

  3. The transformed values are standardized with Huber's M-estimates of location and scale, and those farther from the center than the quantile of the absolute value of a standard normal variable (2.58 for the default 0.99) are given weight zero (first with the rectified transformation of step 2). \lambda is then re-estimated by maximum likelihood on the others. This is repeated nbsteps times.

  4. With standardize = TRUE, the transformed variable is standardized by the mean and standard deviation of these inliers.

With robust = FALSE, \lambda is the classical maximum likelihood estimate (Box and Cox, 1964; Yeo and Johnson, 2000): it maximizes the normal likelihood of all the transformed values, and the transformed variable is standardized by the mean and standard deviation of all the values (quantile and nbsteps are not used). For the Box-Cox transformation, this is the \lambda of MASS::boxcox(). Outliers then pull \lambda toward the value that brings them closer to the other values: the transformed regular values are skewed, and the outliers can be hidden among them (see the examples).

Variables with fewer than 5 values or no spread (zero MAD) are left unchanged. Missing values are ignored by the fit, and stay missing.

Author(s)

Christian L. Goueguel

References

  • Raymaekers, J., Rousseeuw, P.J. (2021). Transforming variables to central normality. Machine Learning, https://doi.org/10.1007/s10994-021-05960-5.

  • Yeo, I.-K., Johnson, R.A. (2000). A new family of power transformations to improve normality or symmetry. Biometrika, 87(4):954-959.

See Also

robust_boxcox(), step_robust_yeojohnson() to use it in a tidymodels recipe, medcouple() to measure the skewness before and after.

Examples

# 300 right-skewed values on both sides of 0 (a gamma distribution shifted
# by -2) and 10 low outliers near -6
set.seed(6)
x <- c(rgamma(300, shape = 2) - 2, rnorm(10, mean = -6, sd = 0.3))
group <- rep(c("regular", "outlier"), c(300, 10))
ggplot2::ggplot(data.frame(x, group), ggplot2::aes(x)) +
  ggplot2::geom_density(fill = "#2166ac", alpha = 0.3) +
  ggplot2::geom_rug(ggplot2::aes(colour = group)) +
  ggplot2::scale_colour_manual(values = c(outlier = "#b2182b", regular = "#2166ac"))

# the low outliers balance the long right tail: the classical fit finds
# lambda close to 1, no transformation; the robust fit compresses the
# right tail (lambda below 1)
classical <- robust_yeojohnson(x, robust = FALSE)
robust <- robust_yeojohnson(x)
round(c(classical = classical$summary$lambda, robust = robust$summary$lambda), 2)

# the classical transformation leaves the regular values skewed to the
# right and the outliers near 3 standard deviations; the robust one makes
# the regular values symmetric and leaves the outliers far out
transformed <- data.frame(
  fit = rep(c("classical", "robust"), each = length(x)),
  value = c(classical$transformation[[1]], robust$transformation[[1]]),
  group = rep(group, 2)
)
ggplot2::ggplot(transformed, ggplot2::aes(value)) +
  ggplot2::geom_density(fill = "#2166ac", alpha = 0.3) +
  ggplot2::geom_rug(ggplot2::aes(colour = group)) +
  ggplot2::scale_colour_manual(values = c(outlier = "#b2182b", regular = "#2166ac")) +
  ggplot2::facet_wrap(~fit, ncol = 1)
regular <- transformed$group == "regular"
# the skewness of the transformed regular values, and the outlier closest
# to them
round(tapply(transformed$value[regular], transformed$fit[regular], medcouple), 2)
round(tapply(transformed$value[!regular], transformed$fit[!regular], max), 2)

data(forageLIBS)
spectra <- snv(forageLIBS[-(1:14)])$correction
wl <- as.numeric(names(spectra))
# the Mg II lines at 279.55 and 280.27 nm: after SNV, most channels have
# negative values in some spectra, which Box-Cox cannot transform
mg <- spectra[which(wl > 279 & wl < 281)]
sum(vapply(mg, min, numeric(1)) < 0)
fit <- robust_yeojohnson(mg[1:300, ])
summary(fit$summary$lambda)
# the skewness of the channels, before and after
summary(sapply(mg[1:300, ], medcouple))
summary(sapply(fit$transformation, medcouple))
# new spectra, with the transformation of the first 300
dim(predict(fit, mg[301:368, ]))

specProc documentation built on Nov. 20, 2026, 5:11 p.m.