agg_scale: Aggregate/Scale a detailed level of a hierarchical variable...

In ihmeuw-demographics/hierarchyUtils: Utility Functions for Hierarchically Structured Data

Aggregate/Scale a detailed level of a hierarchical variable to an aggregate level

Description

Aggregate counts or probabilities from a detailed level of a hierarchical variable to an aggregate level or scale the detailed level values so that the detailed level aggregated together equals the aggregate level.

Usage

agg(
  dt,
  id_cols,
  value_cols,
  col_stem,
  col_type,
  mapping,
  agg_function = sum,
  missing_dt_severity = "stop",
  present_agg_severity = "stop",
  overlapping_dt_severity = "stop",
  na_value_severity = "stop",
  collapse_interval_cols = FALSE,
  quiet = FALSE
)

scale(
  dt,
  id_cols,
  value_cols,
  col_stem,
  col_type,
  mapping = NULL,
  agg_function = sum,
  missing_dt_severity = "stop",
  overlapping_dt_severity = "stop",
  na_value_severity = "stop",
  collapse_interval_cols = FALSE,
  collapse_missing = FALSE,
  quiet = FALSE
)

Arguments

dt	[data.table()] Data to be aggregated or scaled.
id_cols	[character()] ID columns that uniquely identify each row of dt.
value_cols	[character()] Value columns that should be aggregated.
col_stem	[character(1)] The name of the variable to be aggregated or scaled over. If aggregating an 'interval' variable should not include the '_start' or '_end' suffix.
col_type	[character(1)] The type of variable that is being aggregated or scaled over. Can be either 'categorical' or 'interval'.
mapping	[data.table()] For 'categorical' variables, defines how different levels of the hierarchical variable relate to each other. For aggregating 'interval' variables, it is used to specify intervals to aggregate to, while when scaling the mapping is inferred from the available intervals in dt.
agg_function	[⁠function()⁠] Function to use when aggregating, can be either sum (for counts) or prod (for probabilities).
missing_dt_severity	[character(1)] What should happen when dt is missing levels of col_stem that prevent aggregation or scaling from occurring? Can be either 'skip', 'stop', 'warning', 'message', or 'none'. Default is 'stop'. See section on 'Severity Arguments' for more information.
present_agg_severity	[logical(1)] What should happen when dt already has requested aggregates (from mapping)? Can be either 'skip', 'stop', 'warning', 'message', or 'none'. Default is 'stop'. See section on 'Severity Arguments' for more information.
overlapping_dt_severity	[character(1)] When aggregating/scaling an interval variable or collapse_interval_cols=TRUE what should happen when overlapping intervals are identified? Can be either 'skip', 'stop', 'warning', 'message', or 'none'. Default is 'stop'. See section on 'Severity Arguments' for more information.
na_value_severity	[character(1)] What should happen when 'NA' values are present in value_cols? Can be either 'skip', 'stop', 'warning', 'message', or 'none'. Default is 'stop'. See section on 'Severity Arguments' for more information.
collapse_interval_cols	[logical(1)] Whether to collapse interval id_cols (not including col_stem if it is an interval variable). Default is 'False'. If set to 'True' the interval columns are collapsed to the most detailed common intervals and will error out if there are overlapping intervals. See details or vignettes for more information.
quiet	[logical(1)] Should progress messages be suppressed as the function is run? Default is False.
collapse_missing	[logical(1)] When scaling a categorical variable, whether to collapse missing intermediate levels in mapping. Default is 'False' and the function errors out due to missing data.

Details

The agg function can be used to aggregate to different levels of a pre defined hierarchy. For example a categorical variable like location you can aggregate the country level to global level or for a numeric 'interval' variable like age you can aggregate from five year age-groups to all-ages combined.

The scale function can be used to scale different levels of hierarchical variables like location, so that the sub-national level aggregated together equals the national level. Similarly, it can be used to scale a numeric 'interval' variable like age so that the five year age groups aggregated together equals the all-ages value.

If 'location' is the variable to be aggregated or scaled then col_stem = 'location' and 'location' must be included in id_cols. If 'age' is the variable to be aggregated or scaled then col_stem = 'age' and 'age_start' and 'age_end' must be included in id_cols since both variables are needed to represent interval variables.

The mapping argument defines how different levels of the hierarchical variable relate to each other. For numeric interval variables the hierarchy can be inferred while for categorical variables the full hierarchy needs to be provided.

mapping for categorical variables must have columns called 'parent' and 'child' that represent how each possible variable relates to each other. For example if aggregating or scaling locations then mapping needs to define how each child location relates to each parent location. It is then assumed that each parent location in the mapping hierarchy will need to be aggregated to.

mapping for numeric interval variables is only needed when aggregating data to define exactly which aggregates are needed. It must have columns for '{col_stem}_start' and '{col_stem}_end' defining the start and end of each aggregate interval that is need. There can be an optional 'include_NA' logical column that allows 'NA' col_stem values to be included in the aggregate for certain requested aggregates. When scaling data, mapping should be NULL since the hierarchy can be inferred from the available intervals in dt.

agg and scale work even if dt is not a square dataset. Meaning it is okay if different combinations of id_vars have different col_stem values available. For example if making age aggregates, it is okay if some location-years have 5-year age groups while other location-years have 1-year age groups.

If collapse_interval_cols = TRUE it is okay if the interval variables included in id_vars are not all exactly the same, agg and scale will collapse to the most detailed common intervals collapse_common_intervals() prior to aggregation or scaling. An example of this is when aggregating subnational data to the national level (so col_stem is 'location' and col_type is 'categorical') but each subnational location contains different age groups. agg() and scale() first aggregate to the most detailed common age groups before making location aggregates.

The agg and scale functions currently only work when combining counts or probabilities. If the data is in rate-space then you need to convert to count space first, aggregate/scale, and then convert back.

Value

[data.table()] with id_cols and value_cols columns for requested aggregates or with scaled values.

Severity Arguments

missing_dt_severity:

Check for missing levels of col_stem, the variable being aggregated or scaled over.

1. stop: throw error (this is the default).

2. warning or message: throw warning/message and continue with aggregation/scaling for requested aggregations/scalings where expected input data in dt is available.

3. none: don't throw error or warning, continue with aggregation/scaling for requested aggregations/scalings where expected input data in dt is available.

4. skip: skip this check and continue with aggregation/scaling.

present_agg_severity (agg only):

Check for requested aggregates in mapping that are already present

1. stop: throw error (this is the default).

2. warning or message: throw warning/message, drop aggregates and continue with aggregation.

3. none: don't throw error or warning, drop aggregates and continue with aggregation.

4. skip: skip this check and add to the values already present for the aggregates.

na_value_severity:

Check for 'NA' values in the value_cols.

1. stop: throw error (this is the default).

2. warning or message: throw warning/message, drop missing values and continue with aggregation/scaling where possible (this likely will cause another error because of missing_dt_severity, consider setting missing_dt_severity = "skip" for functionality similiar to na.rm = TRUE).

3. none: don't throw error or warning, drop missing values and continue with aggregation/scaling where possible (this likely will cause another error because of missing_dt_severity, consider setting missing_dt_severity = "skip" for functionality similiar to na.rm = TRUE).

4. skip: skip this check and propagate NA values through aggregation/scaling.

overlapping_dt_severity: Check for overlapping intervals that prevent collapsing to the most detailed common set of intervals. Or check for overlapping intervals in col_stem when aggregating/scaling.

1. stop: throw error (this is the default).

2. warning or message: throw warning/message, drop overlapping intervals and continue with aggregation/scaling where possible (this may cause another error because of missing_dt_severity).

3. none: don't throw error or warning, drop overlapping intervals and continue with aggregation/scaling where possible (this may cause another error because of missing_dt_severity).

4. skip: skip this check and continue with aggregation/scaling.

Examples

# aggregate count data from present day Iran provinces to historical
# provinces and Iran as a whole
input_dt <- data.table::CJ(location = iran_mapping[!grepl("[0-9]+", child),
                                                   child],
                           year = 2011,
                           value = 1)
output_dt <- agg(dt = input_dt,
                 id_cols = c("location", "year"),
                 value_cols = "value",
                 col_stem = "location",
                 col_type = "categorical",
                 mapping = iran_mapping)

# scale count data from present day Iran provinces to Iran national value
input_dt <- data.table::CJ(location = iran_mapping[!grepl("[0-9]+", child),
                                                   child],
                           year = 2011,
                           value = 1)
input_dt_agg <- data.table::data.table(
  location = "Iran (Islamic Republic of)",
  year = 2011, value = 62
)
input_dt <- rbind(input_dt, input_dt_agg)
output_dt <- scale(dt = input_dt,
                   id_cols = c("location", "year"),
                   value_cols = "value",
                   col_stem = "location",
                   col_type = "categorical",
                   mapping = iran_mapping,
                   collapse_missing = TRUE)

# aggregate age-specific count data
input_dt <- data.table::data.table(year = 2010,
                        age_start = seq(0, 95, 1),
                        value1 = 1, value2 = 2)
gen_end(input_dt, id_cols = c("year", "age_start"), col_stem = "age")
age_mapping <- data.table::data.table(age_start = c(0, 15, 85),
                                      age_end = c(5, 60, Inf))
output_dt <- agg(dt = input_dt,
                 id_cols = c("year", "age_start", "age_end"),
                 value_cols = c("value1", "value2"),
                 col_stem = "age",
                 col_type = "interval",
                 mapping = age_mapping)

# scale age-specific probability data

ihmeuw-demographics/hierarchyUtils documentation built on June 20, 2024, 7:18 a.m.