auc_compare: Compare AUC values using the FBH method.

In auctestr: Statistical Testing for AUC Data

Description

Apply the FBH method to compare outcome_col by compare_col, averaging over time_col (due to non-independence) and then over over_col by using Stouffer's Method.

Usage

auc_compare(df, compare_values, filter_value, time_col = "time",
  outcome_col = "auc", compare_col = "model_id", over_col = "dataset",
  n_col = "n", n_p_col = "n_p", n_n_col = "n_n",
  filter_col = "model_variant")

Arguments

df	DataFrame containing time_col, outcome_col, compare_col, and over_col.
compare_values	names of models to compare (character vector of length 2). These should match exactly the names as they appear in compare_col.
filter_value	(optional) keep only observations which contain filter_value for filter_col.
time_col	name of column in df representing time of observations (z-scores are averaged over time_col within each model/dataset due to non-independence). These can also be other dependent groupings, such as cross-validation folds.
outcome_col	name of column in df representing outcome to compare; this should be Area Under the Receiver Operating Characteristic or A' statistic (this method applies specifically to AUC and not other metrics (i.e., sensitivity, precision, F1)..
compare_col	name of column in df representing two conditions to compare (should contain at least 2 unique values; these two values are specified as compare_values).
over_col	identifier for independent experiments, iterations, etc. over which z-scores for models are to be compared (using Stouffer's Z).
n_col	name of column in df with total number of observations in the sample tested by each row.
n_p_col	name of column in df with n_p, number of positive observations.
n_n_col	name of column in df with n_n, number of negative observations.
filter_col	(optional) name of column in df to filter observations on; keep only observations which contain filter_value for filter_col.

Value

numeric, overall z-score of comparison using the FBH method.

References

Fogarty, Baker and Hudson, Case Studies in the use of ROC Curve Analysis for Sensor-Based Estimates in Human Computer Interaction, Proceedings of Graphics Interface (2005) pp. 129-136.

Stouffer, S.A.; Suchman, E.A.; DeVinney, L.C.; Star, S.A.; Williams, R.M. Jr. The American Soldier, Vol.1: Adjustment during Army Life (1949).

See Also

Other fbh method: fbh_test, se_auc

Examples

## load sample experiment data
data(sample_experiment_data)
## compare VariantA of ModelA and ModelB
auc_compare(sample_experiment_data,
    compare_values = c('ModelA', 'ModelB'),
    filter_value = c('VariantA'),
    time_col = 'time',
    outcome_col = 'auc',
    compare_col = 'model_id',
    over_col = 'dataset',
    filter_col = 'model_variant')
## compare VariantC of ModelA and ModelB
auc_compare(sample_experiment_data,
    compare_values = c('ModelA', 'ModelB'),
    filter_value = c('VariantC'),
    time_col = 'time',
    outcome_col = 'auc',
    compare_col = 'model_id',
    over_col = 'dataset',
    filter_col = 'model_variant')
## compare ModelC, VariantA and VariantB
auc_compare(sample_experiment_data,
    compare_values = c('VariantA', 'VariantB'),
    filter_value = c('ModelC'),
    time_col = 'time',
    outcome_col = 'auc',
    compare_col = 'model_variant',
    over_col = 'dataset',
    filter_col = 'model_id')

Example output

fetching comparison results for models ModelA, ModelB in dataset dataset1 with filter value VariantA
fetching comparison results for models ModelA, ModelB in dataset dataset2 with filter value VariantA
fetching comparison results for models ModelA, ModelB in dataset dataset3 with filter value VariantA
[1] 0.4298075
Warning message:
`filter_()` is deprecated as of dplyr 0.7.0.
Please use `filter()` instead.
See vignette('programming') for more help
This warning is displayed once every 8 hours.
Call `lifecycle::last_warnings()` to see where this warning was generated. 
fetching comparison results for models ModelA, ModelB in dataset dataset1 with filter value VariantC
fetching comparison results for models ModelA, ModelB in dataset dataset2 with filter value VariantC
fetching comparison results for models ModelA, ModelB in dataset dataset3 with filter value VariantC
[1] 3.604343
fetching comparison results for models VariantA, VariantB in dataset dataset1 with filter value ModelC
fetching comparison results for models VariantA, VariantB in dataset dataset2 with filter value ModelC
fetching comparison results for models VariantA, VariantB in dataset dataset3 with filter value ModelC
[1] 1.655143

auctestr documentation built on May 2, 2019, 2:59 p.m.