new_groupwise_metric | R Documentation |
Groupwise metrics quantify the disparity in value of a metric across a
number of groups. Groupwise metrics with a value of zero indicate that the
underlying metric is equal across groups. yardstick defines
several common fairness metrics using this function, such as
demographic_parity()
, equal_opportunity()
, and equalized_odds()
.
new_groupwise_metric(fn, name, aggregate, direction = "minimize")
fn |
A yardstick metric function or metric set. |
name |
The name of the metric to place in the |
aggregate |
A function to summarize the generated metric set results.
The function takes metric set results as the first argument and returns
a single numeric giving the |
direction |
A string. One of:
|
Note that all yardstick metrics are group-aware in that, when passed
grouped data, they will return metric values calculated for each group.
When passed grouped data, groupwise metrics also return metric values
for each group, but those metric values are calculated by first additionally
grouping by the variable passed to by
and then summarizing the per-group
metric estimates across groups using the function passed as the
aggregate
argument. Learn more about grouping behavior in yardstick using
vignette("grouping", "yardstick")
.
This function is a function factory; it's output is itself a function. Further, the functions that this function outputs are also function factories. More explicitly, this looks like:
# a function with similar implementation to `demographic_parity()`: diff_range <- function(x) {diff(range(x$.estimate))} dem_parity <- new_groupwise_metric( fn = detection_prevalence, name = "dem_parity", aggregate = diff_range )
The outputted dem_parity
is a function that takes one argument, by
,
indicating the data-masked variable giving the sensitive feature.
When called with a by
argument, dem_parity
will return a yardstick
metric function like any other:
dem_parity_by_gender <- dem_parity(gender)
Note that dem_parity
doesn't take any arguments other than by
, and thus
knows nothing about the data it will be applied to other than that it ought
to have a column with name "gender"
in it.
The output dem_parity_by_gender
is a metric function that takes the
same arguments as the function supplied as fn
, in this case
detection_prevalence
. It will thus interface like any other yardstick
function except that it will look for a "gender"
column in
the data it's supplied.
In addition to the examples below, see the documentation on the
return value of fairness metrics like demographic_parity()
,
equal_opportunity()
, or equalized_odds()
to learn more about how the
output of this function can be used.
Additional arguments can be passed to the function outputted by the function that this function outputs. That is:
res_fairness <- new_groupwise_metric(...) res_by <- res_fairness(by) res_by(..., additional_arguments_to_aggregate = TRUE)
For finer control of how groups in by
are treated, use the
aggregate
argument.
data(hpc_cv)
# `demographic_parity`, among other fairness metrics,
# is generated with `new_groupwise_metric()`:
diff_range <- function(x) {diff(range(x$.estimate))}
demographic_parity_ <-
new_groupwise_metric(
fn = detection_prevalence,
name = "demographic_parity",
aggregate = diff_range
)
m_set <- metric_set(demographic_parity_(Resample))
m_set(hpc_cv, truth = obs, estimate = pred)
# the `post` argument can be used to accommodate a wide
# variety of parameterizations. to encode demographic
# parity as a ratio inside of a difference, for example:
ratio_range <- function(x, ...) {
range <- range(x$.estimate)
range[1] / range[2]
}
demographic_parity_ratio <-
new_groupwise_metric(
fn = detection_prevalence,
name = "demographic_parity_ratio",
aggregate = ratio_range
)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.