most_challenging: Find the data points that were hardest to predict

Description Usage Arguments Value Author(s) Examples

View source: R/most_challenging.R

Description

\Sexpr[results=rd, stage=render]{lifecycle::badge("experimental")}

Finds the data points that, overall, were the most challenging to predict, based on a prediction metric.

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
most_challenging(
  data,
  type,
  obs_id_col = "Observation",
  target_col = "Target",
  prediction_cols = ifelse(type == "gaussian", "Prediction", "Predicted Class"),
  threshold = 0.15,
  threshold_is = "percentage",
  metric = NULL,
  cutoff = 0.5
)

Arguments

data

data.frame with predictions, targets and observation IDs. Can be grouped by dplyr::group_by().

Predictions can be passed as values, predicted classes or predicted probabilities:

N.B. Adds .Machine$double.eps to all probabilities to avoid log(0).

Multinomial

When `type` is "multinomial", the predictions can be passed in one of two formats.

Probabilities (Preferable)

One column per class with the probability of that class. The columns should have the name of their class, as they are named in the target column. E.g.:

class_1 class_2 class_3 target
0.269 0.528 0.203 class_2
0.368 0.322 0.310 class_3
0.375 0.371 0.254 class_2
... ... ... ...
Classes

A single column of type character with the predicted classes. E.g.:

prediction target
class_2 class_2
class_1 class_3
class_1 class_2
... ...

Binomial

When `type` is "binomial", the predictions can be passed in one of two formats.

Probabilities (Preferable)

One column with the probability of class being the second class alphabetically ("dog" if classes are "cat" and "dog"). E.g.:

prediction target
0.769 "dog"
0.368 "dog"
0.375 "cat"
... ...

Note: At the alphabetical ordering of the class labels, they are of type character, why e.g. 100 would come before 7.

Classes

A single column of type character with the predicted classes. E.g.:

prediction target
class_0 class_1
class_1 class_1
class_1 class_0
... ...

Gaussian

When `type` is "gaussian", the predictions should be passed as one column with the predicted values. E.g.:

prediction target
28.9 30.2
33.2 27.1
23.4 21.3
... ...
type

Type of task used to get the predictions:

"gaussian" for regression (like linear regression).

"binomial" for binary classification.

"multinomial" for multiclass classification.

obs_id_col

Name of column with observation IDs. This will be used to aggregate the performance of each observation.

target_col

Name of column with the true classes/values in `data`.

prediction_cols

Name(s) of column(s) with the predictions.

threshold

Threshold to filter observations by. Depends on `type` and `threshold_is`.

The threshold can either be a percentage or a score. For percentages, a lower threshold returns fewer observations. For scores, this depends on `type`.

Gaussian

threshold_is "percentage"

(Approximate) percentage of the observations with the largest root mean square errors to return.

threshold_is "score"

Observations with a root mean square error larger than or equal to the threshold will be returned.

Binomial, Multinomial

threshold_is "percentage"

(Approximate) percentage of the observations to return with:

MAE, Cross Entropy: Highest error scores.

Accuracy: Lowest accuracies

threshold_is "score"

MAE, Cross Entropy: Observations with an error score above or equal to the threshold will be returned.

Accuracy: Observations with an accuracy below or equal to the threshold will be returned.

threshold_is

Either "score" or "percentage". See `threshold`.

metric

The metric to use. If NULL, the default metric depends on the format of the prediction columns.

Binomial, Multinomial

"Accuracy", "MAE" or "Cross Entropy".

When one prediction column with predicted classes is passed, the default is "Accuracy". In this configuration, the other metrics are not calculated.

When one or more prediction columns with predicted probabilities are passed, the default is "MAE". This is the Mean Absolute Error of the probability of the target class.

Gaussian

Ignored. Always uses "RMSE".

cutoff

Threshold for predicted classes. (Numeric)

N.B. Binomial only.

Value

data.frame with the most challenging observations and their metrics.

`>=` / `<=` denotes the threshold as score.

Author(s)

Ludvig Renbo Olsen, r-pkgs@ludvigolsen.dk

Examples

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
# Attach packages
library(cvms)
library(dplyr)

##
## Multinomial
##

# Find the most challenging data points (per classifier)
# in the predicted.musicians dataset
# which resembles the "Predictions" tibble from the evaluation results

# Passing predicted probabilities
# Observations with 30% highest MAE scores
most_challenging(
  predicted.musicians,
  obs_id_col = "ID",
  prediction_cols = c("A", "B", "C", "D"),
  type = "multinomial",
  threshold = 0.30
)

# Observations with 25% highest Cross Entropy scores
most_challenging(
  predicted.musicians,
  obs_id_col = "ID",
  prediction_cols = c("A", "B", "C", "D"),
  type = "multinomial",
  threshold = 0.25,
  metric = "Cross Entropy"
)

# Passing predicted classes
# Observations with 30% lowest Accuracy scores
most_challenging(
  predicted.musicians,
  obs_id_col = "ID",
  prediction_cols = "Predicted Class",
  type = "multinomial",
  threshold = 0.30
)

# The 40% lowest-scoring on accuracy per classifier
predicted.musicians %>%
  dplyr::group_by(Classifier) %>%
  most_challenging(
    obs_id_col = "ID",
    prediction_cols = "Predicted Class",
    type = "multinomial",
    threshold = 0.40
  )

# Accuracy scores below 0.05
most_challenging(
  predicted.musicians,
  obs_id_col = "ID",
  type = "multinomial",
  threshold = 0.05,
  threshold_is = "score"
)

##
## Binomial
##

# Subset the predicted.musicians
binom_data <- predicted.musicians %>%
  dplyr::filter(Target %in% c("A","B")) %>%
  dplyr::rename(Prediction = B)

# Passing probabilities
# Observations with 30% highest MAE
most_challenging(
  binom_data,
  obs_id_col = "ID",
  type = "binomial",
  prediction_cols = "Prediction",
  threshold = 0.30
)

# Observations with 30% highest Cross Entropy
most_challenging(
  binom_data,
  obs_id_col = "ID",
  type = "binomial",
  prediction_cols = "Prediction",
  threshold = 0.30,
  metric = "Cross Entropy"
)

# Passing predicted classes
# Observations with 30% lowest Accuracy scores
most_challenging(
  binom_data,
  obs_id_col = "ID",
  type = "binomial",
  prediction_cols = "Predicted Class",
  threshold = 0.30
)

##
## Gaussian
##

set.seed(1)

df <- data.frame(
  "Observation" = rep(1:10, n = 3),
  "Target" = rnorm(n = 30, mean = 25, sd = 5),
  "Prediction" = rnorm(n = 30, mean = 27, sd = 7)
)

# The 20% highest RMSE scores
most_challenging(
  df,
  type = "gaussian",
  threshold = 0.2
)

# RMSE scores above 9
most_challenging(
  df,
  type = "gaussian",
  threshold = 9,
  threshold_is = "score"
)

cvms documentation built on June 17, 2021, 5:12 p.m.