In metrumresearchgroup/bbr: R package for bbi

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

Introduction

This vignette demonstrates how to use summary_log() to extract model diagnostics like the objective function value, condition number, and parameter counts.

If you are new to bbr, the "Getting Started with bbr" vignette will take you through some basic scenarios for modeling with NONMEM using bbr, introducing you to its standard workflow and functionality.

There is a lot of information in the bbi_summary_log_df tibble that is output from summary_log(). However, it is important to note that all of this, and quite a bit more, is contained in the bbi_nonmem_summary object output from model_summary(). If you are trying to dig deep into the outputs of a small number of models, see the Summarize section of the "Getting Started" vignette for an introduction to that functionality.

summary_log() is more useful for getting a slightly higher-level view of a larger batch of models; potentially all the models in a given project, or something like a large group of bootstrapped runs.

Setup

There is some initial set up necessary for using bbr. Please refer to the "Getting Started" vignette, mentioned above, if you have not done this yet. Once this is done, load the library.

# NOTE: if running chunks interactively we need to load the package first
#   because renv isolation prevents us from finding an bbr installation
if (interactive()) {
  devtools::load_all()
}

library(bbr)
library(dplyr)
library(tidyselect)

# set bbi execution path
if (Sys.which("bbi") == "") {
  options('bbr.bbi_exe_path' = read_bbi_path())
}

What's in a Model Summary?

As mentioned above, the bbi_summary_log_df tibble contains a lot of information, which is a subset of what is contained in the bbi_nonmem_summary object returned from model_summary().

MODEL_DIR <- system.file("model", "nonmem", "complex", package = "bbr")

sum1 <- 
  read_model(file.path(MODEL_DIR, "iovmm")) %>%
  model_summary()

names(sum1)

For example, the run_details section alone contains wealth of information about this model run:

str(sum1$run_details)

Much of this is very useful, but it's also a bit intimidating, and it can take some work to unpack it all and find the bits and pieces you're looking for.

What's in the Summary Log?

The summary_log() function is designed to extract some of the most relevant diagnostics and model outputs from a batch of model summaries and organize them into a more easily digestible tibble. Like run_log() and config_log(), it takes two arguments:

.base_dir -- Directory to look for models in.
.recurse -- Logical indicating whether to search recursively in subdirectories. This is TRUE by default.

sum_df <- summary_log(MODEL_DIR)
names(sum_df)

# we're using test data that requires a fail flag, but this is not the focus of this vignette so we hide it.
sum_df <- summary_log(MODEL_DIR, .fail_flags = list(ext_file = "1001.1.TXT"))

The specific columns returned are described below, though there is also a list of them, with brief definitions, in the summary_log() docs that can be accessed any time with ?summary_log() in the console.

Housekeeping Columns

The first column is absolute_model_path which contains an absolute path that unambiguously identifies each model. This serves as the primary key for the tibble. The second column is simply the basename of this path, which is just a convenience for printing and viewing.

The third column contains the bbi_nonmem_summary object, discussed above, for each model. This can be extracted and manipulated if you would like more detailed data from it.

The error_msg and needed_fail_flags columns describe whether bbi had any trouble parsing the model outputs. These won't be discussed in detail here. Refer to the summary_log() docs for more information.

Run Details Columns

The next batch of columns contain the core diagnostics and model outputs. As mentioned above, descriptions of what each column contains can be found in the summary_log() docs.

sum_df %>% 
  collapse_to_string(estimation_method) %>%
  select(
    run,
    ofv, 
    param_count, 
    estimation_method, 
    problem_text, 
    number_of_subjects, 
    number_of_obs, 
    condition_number
  )

Run Heuristics Columns

The run_heuristics element of the bbi_nonmem_summary object contains a number of logical values indicating whether particular heuristic issues were found in the model. Note that these are not necessarily errors with the model run, but are closer to warning flags that should possibly be investigated. Each heuristic is described in more detail in the summary_log() docs.

Note that all heuristics will be FALSE by default (and never NA) and will only be TRUE if they are explicitly triggered. For example, large_condition_number will be FALSE even in the case when a condition number was not calculated at all.

All of the heuristic flags are pivoted out to their own columns in the bbi_summary_log_df tibble. It's useful to note that, except for needed_fail_flags (discussed above), these are the only logical columns in the tibble and can therefore be easily selected with tidyselect::where(is.logical). (Note: where() only became available in tidyselect (>= 1.1.0), released May 2020.)

sum_df %>% select(run, where(is.logical), -needed_fail_flags)

# tidyselect::where() was only added in May 2020 so, in order to avoid having to bump our "oldest supported"
# for the sake of vignette, we just do it the ugly way under-the-hood.
lgl_vars <- purrr::map_lgl(sum_df, is.logical)
sum_df %>% select(c("run", names(lgl_vars)[which(lgl_vars)]))

Notice that there is also an any_heuristics column, which can easily be used to filter to only runs that had at least one heuristic flag triggered.

Add Summary

Just like config_log() has add_config(), you can also use add_summary() to join all of these columns onto an existing bbi_run_log_df (the tibble output from run_log()). This can be useful if you have a run log that you have previously filtered on something like the tags or based_on columns, and you would like to append some simple diagnostics.

# filter to two specific runs
log_df <- run_log(MODEL_DIR, .include = c("acop-iov", "iovmm"))

# add summary columns
log_df <- log_df %>% add_summary()

log_df %>% select(run, ofv, param_count, any_heuristics)