Description Usage Arguments Value References See Also Examples
Apply the FBH method to compare outcome_col
by compare_col
, averaging
over time_col
(due to non-independence) and then over over_col
by
using Stouffer's Method.
1 2 3 4 | auc_compare(df, compare_values, filter_value, time_col = "time",
outcome_col = "auc", compare_col = "model_id", over_col = "dataset",
n_col = "n", n_p_col = "n_p", n_n_col = "n_n",
filter_col = "model_variant")
|
df |
DataFrame containing |
compare_values |
names of models to compare (character vector of length 2). These should match exactly the names as they appear in compare_col. |
filter_value |
(optional) keep only observations which contain
|
time_col |
name of column in df representing time of observations (z-scores are averaged over time_col within each model/dataset due to non-independence). These can also be other dependent groupings, such as cross-validation folds. |
outcome_col |
name of column in df representing outcome to compare; this should be Area Under the Receiver Operating Characteristic or A' statistic (this method applies specifically to AUC and not other metrics (i.e., sensitivity, precision, F1).. |
compare_col |
name of column in df representing two conditions to compare
(should contain at least 2 unique values; these two values are specified as
|
over_col |
identifier for independent experiments, iterations, etc. over which z-scores for models are to be compared (using Stouffer's Z). |
n_col |
name of column in df with total number of observations in the sample tested by each row. |
n_p_col |
name of column in df with n_p, number of positive observations. |
n_n_col |
name of column in df with n_n, number of negative observations. |
filter_col |
(optional) name of column in df to filter observations on; keep only
observations which contain |
numeric, overall z-score of comparison using the FBH method.
Fogarty, Baker and Hudson, Case Studies in the use of ROC Curve Analysis for Sensor-Based Estimates in Human Computer Interaction, Proceedings of Graphics Interface (2005) pp. 129-136.
Stouffer, S.A.; Suchman, E.A.; DeVinney, L.C.; Star, S.A.; Williams, R.M. Jr. The American Soldier, Vol.1: Adjustment during Army Life (1949).
Other fbh method: fbh_test
,
se_auc
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | ## load sample experiment data
data(sample_experiment_data)
## compare VariantA of ModelA and ModelB
auc_compare(sample_experiment_data,
compare_values = c('ModelA', 'ModelB'),
filter_value = c('VariantA'),
time_col = 'time',
outcome_col = 'auc',
compare_col = 'model_id',
over_col = 'dataset',
filter_col = 'model_variant')
## compare VariantC of ModelA and ModelB
auc_compare(sample_experiment_data,
compare_values = c('ModelA', 'ModelB'),
filter_value = c('VariantC'),
time_col = 'time',
outcome_col = 'auc',
compare_col = 'model_id',
over_col = 'dataset',
filter_col = 'model_variant')
## compare ModelC, VariantA and VariantB
auc_compare(sample_experiment_data,
compare_values = c('VariantA', 'VariantB'),
filter_value = c('ModelC'),
time_col = 'time',
outcome_col = 'auc',
compare_col = 'model_variant',
over_col = 'dataset',
filter_col = 'model_id')
|
fetching comparison results for models ModelA, ModelB in dataset dataset1 with filter value VariantA
fetching comparison results for models ModelA, ModelB in dataset dataset2 with filter value VariantA
fetching comparison results for models ModelA, ModelB in dataset dataset3 with filter value VariantA
[1] 0.4298075
Warning message:
`filter_()` is deprecated as of dplyr 0.7.0.
Please use `filter()` instead.
See vignette('programming') for more help
This warning is displayed once every 8 hours.
Call `lifecycle::last_warnings()` to see where this warning was generated.
fetching comparison results for models ModelA, ModelB in dataset dataset1 with filter value VariantC
fetching comparison results for models ModelA, ModelB in dataset dataset2 with filter value VariantC
fetching comparison results for models ModelA, ModelB in dataset dataset3 with filter value VariantC
[1] 3.604343
fetching comparison results for models VariantA, VariantB in dataset dataset1 with filter value ModelC
fetching comparison results for models VariantA, VariantB in dataset dataset2 with filter value ModelC
fetching comparison results for models VariantA, VariantB in dataset dataset3 with filter value ModelC
[1] 1.655143
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.