perf.assess: Compute evaluation criteria for PLS, sPLS, PLS-DA, sPLS-DA,...

In mixOmicsTeam/mixOmics: Omics Data Integration Project

Compute evaluation criteria for PLS, sPLS, PLS-DA, sPLS-DA, MINT and DIABLO

Description

Function to evaluate the performance of the fitted PLS, sparse PLS, PLS-DA, sparse PLS-DA, MINT (mint.splsda) and DIABLO (block.splsda) models using various criteria.

Usage

perf.assess(object, ...)

## S3 method for class 'sgccda'
perf.assess(
  object,
  dist = c("all", "max.dist", "centroids.dist", "mahalanobis.dist"),
  validation = c("Mfold", "loo"),
  folds = 10,
  nrepeat = 1,
  auc = FALSE,
  progressBar = FALSE,
  signif.threshold = 0.01,
  BPPARAM = SerialParam(),
  seed = NULL,
  ...
)

## S3 method for class 'mint.plsda'
perf.assess(
  object,
  dist = c("all", "max.dist", "centroids.dist", "mahalanobis.dist"),
  auc = FALSE,
  progressBar = FALSE,
  signif.threshold = 0.01,
  ...
)

## S3 method for class 'mint.splsda'
perf.assess(
  object,
  dist = c("all", "max.dist", "centroids.dist", "mahalanobis.dist"),
  auc = FALSE,
  progressBar = FALSE,
  signif.threshold = 0.01,
  ...
)

## S3 method for class 'mixo_pls'
perf.assess(
  object,
  validation = c("Mfold", "loo"),
  folds,
  progressBar = FALSE,
  nrepeat = 1,
  BPPARAM = SerialParam(),
  seed = NULL,
  ...
)

## S3 method for class 'mixo_spls'
perf.assess(
  object,
  validation = c("Mfold", "loo"),
  folds,
  progressBar = FALSE,
  nrepeat = 1,
  BPPARAM = SerialParam(),
  seed = NULL,
  ...
)

## S3 method for class 'mixo_plsda'
perf.assess(
  object,
  dist = c("all", "max.dist", "centroids.dist", "mahalanobis.dist"),
  validation = c("Mfold", "loo"),
  folds = 10,
  nrepeat = 1,
  auc = FALSE,
  progressBar = FALSE,
  signif.threshold = 0.01,
  BPPARAM = SerialParam(),
  seed = NULL,
  ...
)

## S3 method for class 'mixo_splsda'
perf.assess(
  object,
  dist = c("all", "max.dist", "centroids.dist", "mahalanobis.dist"),
  validation = c("Mfold", "loo"),
  folds = 10,
  nrepeat = 1,
  auc = FALSE,
  progressBar = FALSE,
  signif.threshold = 0.01,
  BPPARAM = SerialParam(),
  seed = NULL,
  ...
)

Arguments

object	object of class inherited from "pls", "plsda", "spls", "splsda". "sgccda" or "mint.splsda". The function will retrieve some key parameters stored in that object.
...	not used
dist	only applies to an object inheriting from "plsda", "splsda" or "mint.splsda" to evaluate the classification performance of the model. Should be a subset of "max.dist", "centroids.dist", "mahalanobis.dist". Default is "all". See predict.
validation	a character string. What kind of (internal) validation to use, matching one of "Mfold" or "loo" (see below). Default is "Mfold". For MINT methods only "loo" will be used.
folds	numeric. Number of folds in the Mfold cross-validation. See Details.
nrepeat	numierc. Number of times the Cross-Validation process is repeated. This is an important argument to ensure the estimation of the performance to be as accurate as possible. Default it 1.
auc	if TRUE calculate the Area Under the Curve (AUC) performance of the model.
progressBar	by default set to FALSE to output the progress bar of the computation.
signif.threshold	numeric between 0 and 1 indicating the significance threshold required for improvement in error rate of the components. Default to 0.01.
BPPARAM	A BiocParallelParam object indicating the type of parallelisation. See examples in ?tune.spca.
seed	set a number here if you want the function to give reproducible outputs. Not recommended during exploratory analysis. Note if RNGseed is set in 'BPPARAM', this will be overwritten by 'seed'. Note 'seed' is not required or used in perf.mint.plsda as this method uses loo cross-validation

Details

This function is built upon 'perf()' but instead of assessing model performance across components 1:ncomp only assesses performance of the given model

Procedure. The process of evaluating the performance of a fitted model object is similar for all PLS-derived methods; a cross-validation approach is used to fit the method of object on folds-1 subsets of the data and then to predict on the subset left out. Different measures of performance are available depending on the model. Parameters such as logratio, multilevel, keepX or keepY are retrieved from object.

Parameters. If validation = "Mfold", M-fold cross-validation is performed. folds specifies the number of folds to generate. The folds also can be supplied as a list of vectors containing the indexes defining each fold as produced by split. When using validation = "Mfold", make sure that you repeat the process several times (as the results will be highly dependent on the random splits and the sample size).

If validation = "loo", leave-one-out cross-validation is performed (in that case, there is no need to repeat the process).

Measures of performance. For fitted PLS and sPLS regression models, perf estimates the mean squared error of prediction (MSEP), R^2, and Q^2 to assess the predictive perfity of the model using M-fold or leave-one-out cross-validation. Note that only the classic, regression and invariant modes can be applied. For sPLS, the MSEP, R^2, and Q^2 criteria are averaged across all folds. Note that for PLS and sPLS objects, perf is performed on the pre-processed data after log ratio transform and multilevel analysis, if any.

The mint.sPLS-DA function estimates errors based on Leave-one-group-out cross validation (where each levels of object$study is left out (and predicted) once) and provides study-specific outputs (study.specific.error) as well as global outputs (global.error). Note the mint perf methods do not use seed or BPPARAM arguments.

AUROC. For PLS-DA, sPLS-DA, mint.PLS-DA, mint.sPLS-DA, and block.splsda methods: if auc=TRUE, Area Under the Curve (AUC) values are calculated from the predicted scores obtained from the predict function applied to the internal test sets in the cross-validation process, either for all samples or for study-specific samples (for mint models). Therefore we minimise the risk of overfitting. For block.splsda model, the calculated AUC is simply the blocks-combined AUC calculated using auroc.sgccda. See auroc for more details. Our multivariate supervised methods already use a prediction threshold based on distances (see predict) that optimally determine class membership of the samples tested. As such AUC and ROC are not needed to estimate the performance of the model. We provide those outputs as complementary performance measures.

Prediction distances. See details from ?predict, and also our supplemental material in the mixOmics article.

Repeats of the CV-folds. Repeated cross-validation implies that the whole CV process is repeated a number of times (nrepeat) to reduce variability across the different subset partitions. In the case of Leave-One-Out CV (validation = 'loo'), each sample is left out once (folds = N is set internally) and therefore nrepeat is by default 1.

BER is appropriate in case of an unbalanced number of samples per class as it calculates the average proportion of wrongly classified samples in each class, weighted by the number of samples in each class. BER is less biased towards majority classes during the performance assessment.

For sgccda objects, we provide weighted measures (e.g. error rate) in which the weights are simply the correlation of the derived components of a given block with the outcome variable Y.

More details about the PLS modes in ?pls.

Value

For PLS and sPLS models:

MSEP	Mean Square Error Prediction for each Y variable, only applies to object inherited from "pls", and "spls". Only available when in regression (s)PLS.
RMSEP	Root Mean Square Error Prediction for each Y variable, only applies to object inherited from "pls", and "spls". Only available when in regression (s)PLS.
R2	a matrix of R^2 values of the Y-variables for models with 1, \ldots ,ncomp components, only applies to object inherited from "pls", and "spls". Only available when in regression (s)PLS.
Q2	if Y contains one variable, a vector of Q^2 values else a list with a matrix of Q^2 values for each Y-variable. Note that in the specific case of an sPLS model, it is better to have a look at the Q2.total criterion, only applies to object inherited from "pls", and "spls". Only available when in regression (s)PLS.
Q2.total	a vector of Q^2-total values for models with 1, \ldots ,ncomp components, only applies to object inherited from "pls", and "spls". Available in both (s)PLS modes.
RSS	Residual Sum of Squares across all selected features
PRESS	Predicted Residual Error Sum of Squares across all selected features
cor.tpred, cor.upred	Correlation between the predicted and actual components for X (t) and Y (u)
RSS.tpred, RSS.upred	Residual Sum of Squares between the predicted and actual components for X (t) and Y (u)

For PLS-DA and sPLS-DA models:

error.rate	Prediction error rate for each dist and measure
auc	AUC value averaged over the nrepeat
auc.all	AUC values per repeat
predict	Predicted values of each sample for each class
class	A list which gives the predicted class of each sample for each dist and each of the ncomp components

For mint.splsda models:

study.specific.error	A list that gives BER, overall error rate and error rate per class, for each study
global.error	A list that gives BER, overall error rate and error rate per class for all samples
predict	A list of length ncomp that produces the predicted values of each sample for each class
class	A list which gives the predicted class of each sample for each dist.
auc	AUC values
auc.study	AUC values for each study in mint models

For sgccda models (i.e. block (s)PLS-DA models):

error.rate	Prediction error rate for each block of object$X and each dist
error.rate.per.class	Prediction error rate for each block of object$X, each dist and each class
predict	Predicted values of each sample for each class and each block
class	Predicted class of each sample for each block, each dist, and each nrepeat
AveragedPredict.class	if more than one block, returns the average predicted class over the blocks (averaged of the Predict output and prediction using the max.dist distance)
AveragedPredict.error.rate	if more than one block, returns the average predicted error rate over the blocks (using the AveragedPredict.class output)
WeightedPredict.class	if more than one block, returns the weighted predicted class over the blocks (weighted average of the Predict output and prediction using the max.dist distance). See details for more info on weights.
WeightedPredict.error.rate	if more than one block, returns the weighted average predicted error rate over the blocks (using the WeightedPredict.class output.)
MajorityVote	if more than one block, returns the majority class over the blocks. NA for a sample means that there is no consensus on the predicted class for this particular sample over the blocks.
MajorityVote.error.rate	if more than one block, returns the error rate of the MajorityVote output
WeightedVote	if more than one block, returns the weighted majority class over the blocks. NA for a sample means that there is no consensus on the predicted class for this particular sample over the blocks.
WeightedVote.error.rate	if more than one block, returns the error rate of the WeightedVote output
weights	Returns the weights of each block used for the weighted predictions, for each nrepeat and each fold

Author(s)

Ignacio González, Amrit Singh, Kim-Anh Lê Cao, Benoit Gautier, Florian Rohart, Al J Abadi

References

Singh A., Shannon C., Gautier B., Rohart F., Vacher M., Tebbutt S. and Lê Cao K.A. (2019), DIABLO: an integrative approach for identifying key molecular drivers from multi-omics assays, Bioinformatics, Volume 35, Issue 17, 1 September 2019, Pages 3055–3062.

mixOmics article:

Rohart F, Gautier B, Singh A, Lê Cao K-A. mixOmics: an R package for 'omics feature selection and multiple data integration. PLoS Comput Biol 13(11): e1005752

MINT:

Rohart F, Eslami A, Matigian, N, Bougeard S, Lê Cao K-A (2017). MINT: A multivariate integrative approach to identify a reproducible biomarker signature across multiple experiments and platforms. BMC Bioinformatics 18:128.

PLS and PLS citeria for PLS regression: Tenenhaus, M. (1998). La regression PLS: theorie et pratique. Paris: Editions Technic.

Chavent, Marie and Patouille, Brigitte (2003). Calcul des coefficients de regression et du PRESS en regression PLS1. Modulad n, 30 1-11. (this is the formula we use to calculate the Q2 in perf.pls and perf.spls)

Mevik, B.-H., Cederkvist, H. R. (2004). Mean Squared Error of Prediction (MSEP) Estimates for Principal Component Regression (PCR) and Partial Least Squares Regression (PLSR). Journal of Chemometrics 18(9), 422-429.

sparse PLS regression mode:

Lê Cao, K. A., Rossouw D., Robert-Granie, C. and Besse, P. (2008). A sparse PLS for variable selection when integrating Omics data. Statistical Applications in Genetics and Molecular Biology 7, article 35.

One-sided t-tests (suppl material):

Rohart F, Mason EA, Matigian N, Mosbergen R, Korn O, Chen T, Butcher S, Patel J, Atkinson K, Khosrotehrani K, Fisk NM, Lê Cao K-A&, Wells CA& (2016). A Molecular Classification of Human Mesenchymal Stromal Cells. PeerJ 4:e1845.

See Also

predict, nipals, plot.perf, auroc and www.mixOmics.org for more details.

Examples

## PLS-DA example

data(liver.toxicity) # rats gex and clinical measurements/treatments
unique(liver.toxicity$treatment$Treatment.Group) # 16 groups
length(liver.toxicity$treatment$Treatment.Group) # 64 samples

plsda.res <- plsda(liver.toxicity$gene, liver.toxicity$treatment$Treatment.Group, ncomp = 2)

performance <- perf.assess(plsda.res, 
                        # to make sure each fold has all classes represented
                           validation = "Mfold", folds = 3, nrepeat = 10, 
                           seed = 12) # for reproducibility, remove for analysis

performance$error.rate$BER

## sPLS-DA example

splsda.res <- splsda(liver.toxicity$gene, liver.toxicity$treatment$Treatment.Group,
                     keepX = c(25, 25), ncomp = 2)

performance <- perf.assess(splsda.res,
                           validation = "Mfold", folds = 3, nrepeat = 10, 
                           seed = 12)

performance$error.rate$BER # can see slight improvement in error rate over PLS-DA example

## PLS example

ncol(liver.toxicity$clinic) # 10 Y variables as output of PLS model

pls.res <- pls(liver.toxicity$gene, liver.toxicity$clinic, ncomp = 2)

performance <- perf.assess(pls.res, 
                           validation = "Mfold", folds = 3, nrepeat = 10,
                           seed = 12)

# see Q2 which gives indication of predictive ability for each of the 10 Y outputs
performance$measures$Q2$summary 

## sPLS example

spls.res <- spls(liver.toxicity$gene, liver.toxicity$clinic, ncomp = 2, keepX = c(50, 50))

performance <- perf.assess(spls.res, 
                           validation = "Mfold", folds = 3, nrepeat = 10, 
                           seed = 12)

# see Q2 which gives indication of predictive ability for each of the 10 Y outputs
performance$measures$Q2$summary


## block PLS-DA example

data("breast.TCGA")
mrna <- breast.TCGA$data.train$mrna
mirna <- breast.TCGA$data.train$mirna
data <- list(mrna = mrna, mirna = mirna)
design <- matrix(1, ncol = length(data), nrow = length(data), dimnames = list(names(data), names(data)))
diag(design) <-  0

block.plsda.res <- block.plsda(X = data, Y = breast.TCGA$data.train$subtype, 
                               ncomp = 2, design = design)

performance <- perf.assess(block.plsda.res)

performance$error.rate.per.class # error rate per class per distance metric

## block sPLS-DA example

block.splsda.res <- block.splsda(X = data, Y = breast.TCGA$data.train$subtype, 
                                 ncomp = 2, design = design,
                                 keepX = list(mrna = c(8,8), mirna = c(8,8)))

performance <- perf.assess(block.splsda.res)

performance$error.rate.per.class

## MINT PLS-DA example

data("stemcells")

mint.plsda.res <- mint.plsda(X = stemcells$gene, Y = stemcells$celltype, ncomp = 3,
                             study = stemcells$study)

performance <- perf.assess(mint.plsda.res)

performance$global.error$BER # global error per distance metric

## MINT sPLS-DA example

mint.splsda.res <- mint.splsda(X = stemcells$gene, Y = stemcells$celltype, ncomp = 3, 
                               keepX = c(10, 5, 15), study = stemcells$study)

performance <- perf.assess(mint.splsda.res)

performance$global.error$BER # error slightly higher in this sparse model verses non-sparse

mixOmicsTeam/mixOmics documentation built on Dec. 3, 2024, 11:15 p.m.