knitr::opts_chunk$set(out.width = "100%", warning = FALSE, message = FALSE)

theophylline <- file.path(system.file(package = "ggPMX"), "testdata", "theophylline")
work_dir <- file.path(theophylline, "Monolix")
input_data <- file.path(theophylline, "data_pk.csv")

ctr <- theophylline()


The ggPMX package generates standard diagnostic plots and tables for mixed effect models used in Pharmacometric (PMX) activities. The tool is built upon the ggplot2 package and is planned to support models developped either with Monolix or nlmixr software. The current release ( supports models fitted with Monolix versions 2016 and later.

The package aims to provide a workflow that is consistent, efficient and which results in high quality graphics ready to use in official documents and reports. The package allows a high degree of flexibility and customization, yet providing an acceptable default setting. The package also allows to fully automate plots and report generation.

The general context is the analysis of mixed effect models fitted to data. ggPMX was developed in the framework of Pharmacometric activities, in which case the model is a population pharmacokinetic (PK) and/or pharmacodynamic (PD) model and the data is clinical or pre-clinical PK and/or PD data.

In the context of model building, evaluation and qualification, it is good practice to assess the goodness-of-fit of models by inspecting (qualitatively and quantitatively) a set of graphs that indicate how well the model describes the data. Several types of diagnostic plots allow to evaluate a mixed effects model fit, the most common being:

a. residual-based plots b. prediction-based plots c. empirical Bayes estimates (EBE)-based plots d. simulation-based plots.

The following figures are examples of diagnotic plots.

ctr <- theophylline()
ctr %>% pmx_plot_dv_pred
ctr %>% pmx_plot_npde_time
ctr %>% pmx_plot_vpc
ctr %>% pmx_plot_eta_box
ctr %>% pmx_plot_eta_matrix(

This document introduces the ggPMX functionalities and syntax.


The high level architecture is represented in the figure below. The key components of the package are the following:


The package is coded using object-oriented programming meaning that information is encoded in objects. The user can change or print the content of objets via functions. Such an implementation allows to have code that is modular and easily customizable.

Workflow overview

The typical workflow of ggPMX is as follows:

  1. The user creates the Controller using pre-defined configurations (yaml templates) for plot settings.
  2. The Controller implicitly calls the Reader that reads and stores modelling outputs into a standard format. As a result, the Controller contains all available plots with their default values according to the configuration used.
  3. The Generator allows to print the available plots by calling the corresponding functions. Plots can be modified by using optional arguments.
  4. A call to the Reporter allows to create a pdf or docx report. The report Rmarkdown template can also be personalized.

The most important task for the user is the Controller creation. This step requires careful consideration because it involves different options according to the type of model (PK or PKPD) and software (Monolix or nlmixr) used for model fitting. The next section describes the Controller creation for the different possible cases.

Once the Controller is created, it implicitly calls the Reader and creates the diagnostic plots. The user can then generate the graphs by calling pre-defined functions as described in Section 3. The same syntax is used independent of the model structure (PK or PKPD model) and of the fitting software.

The Reporter creates one report per endpoint, i.e., one report for PK and one for each PD endpoint.

Modeling datasets

For the sake of this document, three types of datasets are defined.


A diagnostic session starts with the creation of a Controller. The Controller is the "user interface" of the package and allows to conrol all possible options. It is a container that stores configuration fields (model- and input data-related information), datasets and plots. It can be used as a reference object in which the user can see the names of the exisitng plots, the names of the ggPMX datasets, etc. The syntax of the Controller creation differs depending on the software used for model fitting and on the number of model endpoints (or outputs). This section presents different cases of Controller creation. For simplicity, the case of models with one single output is presented first (Section 2.1), then generalized to several outputs (Section 2.2). In Section 2.1 are presented other Controller creation functions that can be used with the different fitting softwares. Note that all these functions can also be used with models with several outputs. Section 2.7 provides details on the content of the Controller.

Single-endpoint models

In general, models with only one endpoint (or output) are mostly PK models, but these could also be k-PD models.

To illustrate ggPMX functionalities, the single-endpoint built-in model called theophylline is used hereafter. The theophylline population PK example has the following characteristics:

The input modeling dataset has the following columns:

theophylline_path <- file.path(system.file(package = "ggPMX"), "testdata", "theophylline")
work_dir          <- file.path(theophylline_path, "Monolix")
input_data_path   <- file.path(theophylline_path, "data_pk.csv")

input_data_theo   <- read.csv(input_data_path)

Note that the DVID (or CMT/YTYPE) column is missing, but since this is a single-endpoint model, it is not necessary in that case.

Generic Controller creation with pmx()

The function pmx() is the generic function for greating a Controller. The user needs to specify a set of arguments such as the path to the model directory, the software used for model fitting (Monolix or nlmixr), the name of a configuration. A list of all existing configurations is provided in the Appendix. All mandatory arguments of pmx() are listed in Table \ref{tab:pmx_mandatory}.

out <- rbind(
  c("sys", "Software used for model fittng (Monolix or nlmixr)", "mlx, mlx2018, nm"),
  c("config", "A pre-defined configuration is a set of default settings", "standing"),
  c("directory", "Path to the directory containing model output files", ""),
  c("input", "Path to input modeling dataset (dataset used for model fitting)", ""),
  c("dv", "Measurable variable name, as defined in the input modeling dataset", "DV, LIDV, LNDV, Y, etc."),
  c("dvid", "Endpoint (output) name, as defined in the input modeling dataset", "DVID, YTYPE, CMT, etc.")

colnames(out) <- c("Argument", "Description", "Values")

xt <- xtable(head(out), label = "tab:pmx_mandatory", caption = "Mandatory arguments of pmx() function")
print(xt, comment = F)

The example below defines a Controller with the standing (standard) configuration.

theophylline_path <- file.path(system.file(package = "ggPMX"), "testdata", "theophylline")
work_dir          <- file.path(theophylline_path, "Monolix")
input_data_path   <- file.path(theophylline_path, "data_pk.csv")

ctr <- pmx(
  sys       = "mlx",
  config    = "standing",
  directory = work_dir,
  input     = input_data_path,
  dv        = "Y",
  dvid      = "DVID"

Note that the column "DVID" of data_pk.csv does not exist; however it is not needed here because there is only one single output of the model. As dvid is a mandatory argument, it still needs to be provided and was set arbritrarly to "DVID" in the example above.

The input dataset can be provided to ggPMX via its location (as in the example above) or as a data frame (maybe give an example). The modeling output datasets have to be in the location that is indicated as working directory (work_dir in the example above).

The above example of Contoller creation is wrapped in a function called "theophylline()" for quick reference:

ctr <- theophylline()

The following are optional arguments to the pmx() function (for details of each option, see the corresponding section):

Models fitted with Monolix (versions 2016 and later)


The controller initialization can be simplified by using the Monolix controller pmx_mlx(), which is a wrapper function for pmx() with sys="mlx". Note that the sys argument is no longer required.

ctr <- pmx_mlx(
  config    = "standing",
  directory = work_dir,
  input     = input_data_path,
  dv        = "Y",
  dvid      = "DVID"


The controller initialization can be simplified even further by using the Monolix controller pmx_mlxtran(). This function parses the mlxtran file of a Monolix project and assigns automatically the different fields necessary to the Controller creation. The only mandatory argument is file_name, the path to the mlxtran file.

mlxtran_path <- file.path(system.file(package = "ggPMX"), 
                          "testdata", "1_popPK_model", "project.mlxtran")

ctr <- pmx_mlxtran(file_name = mlxtran_path)

The user can verify the content of the Controller and how parameters are assigned by printing it (see Section 2.4).

Multiple-endpoint models

Models with more than one endpoint (or output) are mostly PKPD models, but these could also be, for example, PK binding models in which there are measurements and predictions of both PK and its target.

ggPMX produces one diagnostics report per endpoint. As a consequence, the endpoint (if more than one) should be set at the time of the Controller creation in order to filter the observations dataset and to keep only the values corresponding to the endpoint of interest. To handle this, the user creates an "endpoint" object using the function pmx_endpoint() having the following attributes:

To illustrate the Controller creation with multiple-endpoint models, a built-in PKPD example is used. The input dataset is called pk_pd.csv and has the following columns.

pkpd_path       <- file.path(system.file(package = "ggPMX"), "testdata", "pk_pd")
pkpd_work_dir   <- file.path(pkpd_path, "RESULTS")
pkpd_input_file <- file.path(pkpd_path, "pk_pd.csv")

input_data <- read.csv(pkpd_input_file)

The dvid column contains values=3 for PK (first endpoint) and dose and =4 for PD (second endpoint). Monolix2016 outputs are found in folder RESULTS/ which contains predictions1.txt and finegrid1.txt for PK predictions, and predictions2.txt and finegrid2.txt for PD predictions. The Endpoint and Controller objects are created as follows:

pkpd_path       <- file.path(system.file(package = "ggPMX"), "testdata", "pk_pd")
pkpd_work_dir   <- file.path(pkpd_path, "RESULTS")
pkpd_input_file <- file.path(pkpd_path, "pk_pd.csv")

ep <- pmx_endpoint(
  code  = "4",
  label = "some_label",
  unit  = "some_unit",
  file.code = "2", # will use predictions2.txt and finegrig2.txt
  trans = "log10"

ctr <- pmx_mlx(
  config    = "standing",
  directory = pkpd_work_dir,
  input     = pkpd_input_file,
  dv        = "dv",
  dvid      = "dvid",
  endpoint  = ep

A simplified syntax for the Endpoint creation exists in the case where the endpoint code matches the files post-fixes (code=1 corresponds to predictions1.txt, code=2 corresponds to predictions2.txt). Instead of passing a pmxEndpoint object as argument of the Controller, the user can specify the numerical value corresponding to the YTYPE/CMT/DVID column.

  dvid = "YTYPE", ## use this column as obseration id 
  endpoint = 1,   ## select the first endpoint 
  ...)            ## other pmx parameters , config, input,etc..

Internally, a pmxEndpoint object will be created, and observations having YTYPE=x will be filered.

Controller with covariates

Besides the mandatory fields to initialize a Controller, the user can set optional parameters related to covariates. This feature is illustrated below with the Theophylline example.

theophylline_path <- file.path(system.file(package = "ggPMX"), "testdata", "theophylline")
work_dir          <- file.path(theophylline_path, "Monolix")
input_data_path   <- file.path(theophylline_path, "data_pk.csv")

ctr <- pmx_mlx(
  config    = "standing",
  directory = work_dir,
  input     = input_data_path,
  dv        = "Y",
  dvid      = "DVID",
  cats      = c("SEX"),
  conts     = c("WT0", "AGE0"),
  strats    = c("STUD", "SEX")

Conts are the continuous covariates. Cats are categorical covariates used in the model, whereas strats are categorical variables that can be used for plot stratification, but are not used as covariates in the model.

The covariates can be accessed using helper functions:

ctr %>% get_cats()
ctr %>% get_conts()
ctr %>% get_strats()
ctr %>% get_covariates()

Controller content

The content of the Controller can be seen by printing it:


It contains three tables:

Plot names

The Controller is a container that stores all plots. To get the list of plots, the function plot_names() is used:

ctr %>% plot_names()

An alternative way to display the names of the existing plots is by printing the content of the Controller as done above.

ggPMX provides a specialized function to create and update each plot pmx_plot_xx() where xx is the plot name from the list above. These functions are described in detail in Section 3.

Plot types

Each plot type is a class of similar plots. ggPMX defines the following plot types:

The following syntax allows to see which type of plot corresponds to which plot name:

ctr %>% plots()


ggPMX uses the following dataset name convention. The input modeling dataset is the one used for model fitting (the actual data). The output modeling datasets are those output from the fitting tool (Monolix or NONMEM). The ggPMX datasets are the ones created within ggPMX. Table \ref{tab:ggPMX_datasets} provides a list of all ggPMX datasets.

out <- rbind(
  c("input", "Input modeling dataset"),
  c("estimates", "Estimated population parameters"),
  c("eta", "Random effects, their standard deviation and residual errors (to calculate shrinkage)"),
  c("predictions", "Observations and predictions at times of observations dataset"),
  c("finegrid", "Additional predictions (at times without observations)")

colnames(out) <- c("ggPMX dataset", "Description")
# knitr::kable(out)
# latex(head(out), file='', label='tab:ggPMX_datasets', caption='ggPMX datasets',where = "!htbp")
xt <- xtable(head(out), label = "tab:ggPMX_datasets", caption = "ggPMX datasets")
print(xt, comment = F)

Default diagnostic plots

theophylline <- file.path(system.file(package = "ggPMX"), "testdata", "theophylline")
work_dir <- file.path(theophylline, "Monolix")
input_data <- file.path(theophylline, "data_pk.csv")

ctr <- theophylline()

The diagnostic plots of ggPMX are generated by calling the functions pmx_plot_xx() where xx is a placeholder for the plot name. The list of names of all available plots can be seen via:

ctr %>% plot_names()

As a convention, when plots are described as ???Y vs. X???, it is meant that Y is plotted on the vertical axis and X on the horizontal axis.

As an example, a plot of individual weighted residuals (IWRES) versus time (with default settings) can be generated using a single-line code:

ctr %>% pmx_plot_iwres_time

The complete list of available plots per plot type (given in parenthesis) is given below:

ctr %>% pmx_plot_dv_pred
ctr %>% pmx_plot_dv_ipred

ctr %>% pmx_plot_iwres_time
ctr %>% pmx_plot_npde_time

ctr %>% pmx_plot_iwres_ipred
ctr %>% pmx_plot_abs_iwres_ipred

ctr %>% pmx_plot_npde_pred
ctr %>% pmx_plot_eta_hist
ctr %>% pmx_plot_eta_box
ctr %>% pmx_plot_individual(npage = 1)
ctr %>% pmx_plot_npde_qq
ctr %>% pmx_plot_iwres_qq
ctr %>% pmx_plot_eta_matrix

Visual Predictive Checks (VPC)


pmx_sim creates a simulation object. It takes the following arguments:


  1. file character path to the simulation file
  2. irun character name of the simulation column
  3. idv character name of the ind. variable
  4. dv character name of the observation variable

Within pmx vpc controller, it is called like :

theoph_path <- file.path(
  system.file(package = "ggPMX"), "testdata",
WORK_DIR <- file.path(theoph_path, "Monolix")
input_file <- file.path(theoph_path, "data_pk.csv")
vpc_file <- file.path(theoph_path, "sim.csv")

ctr <- pmx_mlx(
  config = "standing",
  directory = WORK_DIR,
  input = input_file,
  dv = "Y",
  cats = c("SEX"),
  conts = c("WT0", "AGE0"),
  strats = "STUD",
  settings = pmx_settings(
  sim = pmx_sim(
    file = vpc_file,
    irun ="rep",

VPC plot

The plot options are described in ?pmx_plot_vpc function.


ctr %>% pmx_plot_vpc


By default the vpc plot is percentile ; , but we can plot the scatter type:

ctr %>% pmx_plot_vpc(type ="scatter")


ctr %>% pmx_plot_vpc(bin=pmx_vpc_bin(style = "kmeans",n=5))


ctr %>% pmx_plot_vpc(strat.facet="SEX",facets=list(nrow=2))

Monolix-like customisation

User can customize the options to get a Monolix-like display.

ctr %>% pmx_plot_vpc(
  is.draft = FALSE,
  pi = pmx_vpc_pi(interval = c(0.1,0.9),
              extreme= list(color="green")),
  obs = pmx_vpc_obs(color="blue",shape=18,size=2),
  ci = pmx_vpc_ci(interval = c(0.1,0.9),

Diagnostics report

theophylline <- file.path(system.file(package = "ggPMX"), "testdata", "theophylline")
work_dir <- file.path(theophylline, "Monolix")
input_data <- file.path(theophylline, "data_pk.csv")

ctr <- theophylline()

A report (in pdf and docx format) containing all default diagnostic plots can be created using the pmx_report function. The format can take three different values:


ctr %>% pmx_report(name='Diagnostic_plots2',
                   save_dir = work_dir,

Note that running the same command first with the option "format='plots'" and then with the option "format='report'" will remove the ggpmx_GOF folder.

Note also that by default, the report will have the DRAFT label on all plots. The label can be removed by using the settings argument in the Controller creation, as described in Section 6.3.1.

The user can customize the default report by creating a "template". To create a template, the user should create first a default report with the following command:

ctr %>% pmx_report(name='Diagnostic_plots1',
                   save_dir = work_dir,

The Rmarkdown (.Rmd) file is the "template". The user can modify the Rmarkdown file as desired (ex: changing the size of some figures) and save the modified file. The new template can be used with the following command:

ctr %>% pmx_report(name='Diagnostic_plots3',
                   save_dir = work_dir,

Customizing plots

Any particular plot can be customized in two ways:

ctr %>% pmx_plot_xx(list of options)
ctr %>% pmx_update(???xx???, list of options)

Help(pmx_gpar) lists some options.

Help(pmx_plot_xx) lists some possible parameters to update a particular plot.

To obtain an exhaustive list of possible options for a particular plot, use the following:

ctr %>% get_plot_config("xx")


ctr = theophylline()
bloq = list(show = FALSE, color = "blue", alpha = 0.2, size = 3, pch = 8)

Customizing global settings - pmx_settings()

The user can define a Controller with global settings that will be applied to all plots. For example remove draft annoataion, use abbreviation defintions to define axis labels, etc.

A settings object is defined by using the function pmx_settings(). The created object is passed as the parameter "settings" to pmx(). By doing so, the settings are defined globally and apply to all plots. For a complete list of global settings with their corresponding default values, please consult the ggPMX Help (?pmx_settings).

## set one or more settings
my_settings <- pmx_settings(
  is.draft   = FALSE,
  use.abbrev = TRUE,
  ...) ### set other settings parameters here 
ctr <-
    ..., ## put here other pmx parametes 
    settings = my_settings

Remove DRAFT label globally

By default in the "standing" configuration, a DRAFT label is printed on all plots. In order to switch this label off, the user sets the is.draft option of pmx_settings() to FALSE.

ctr <- theophylline(settings = pmx_settings(is.draft = FALSE))

Use abbreviation definitions

The standing configuration initializes all plots using abbreviations for axis labels. Each abbreviation has its corresponding definition. To get the list of abbreviation :

ctr %>% get_abbrev

You can update one abbreviation to set a custom label

ctr %>% set_abbrev(TIME="TIME after the first dose")

Using use.abbrev flag you can use abbreviation definition to set axis labels:

ctr <- theophylline(settings=pmx_settings(use.abbrev = TRUE))
ctr %>% set_abbrev(TIME="Custom TIME axis")
ctr %>% pmx_plot_npde_time

Use finegrid.txt file for individual plots

within Monolix, user can choose to not use finegrid file even if it is present.

ctr <- theophylline()
ctr %>% pmx_plot_individual(use.finegrid =FALSE)

Set stratification color legend

In case of color startfication user can customize the legend. For example here using the ggplot2::scale_color_manual:

ctr <- theophylline()
ctr %>% pmx_plot_npde_time(strat.color="STUD")+ 
        labels=c("Study 1","Study 2"),

Another way to do it is to define a global scales.color parameter that will applied to all plots with strat.color :

ctr <- theophylline(
        labels=c("Study 1","Study 2"),

ctr %>% pmx_plot_npde_time(strat.color="STUD")
ctr %>% pmx_plot_eta_box(strat.color="STUD")

Define labels of categorical variables

In case of faceting by stratification user can redfine categorical labels to have more human readables strips. Lables are defined within cats.labels argument and user can use them by setting use.lables to TRUE.

ctr <- theophylline(
        STUD=c("1"="Study 1","2"="Study 2")
      use.labels = TRUE

ctr %>% pmx_plot_npde_time(strat.facet=~SEX)
ctr <- theophylline(
        STUD=c("1"="Study 1","2"="Study 2")
      use.labels = TRUE

ctr %>% pmx_plot_npde_time(strat.facet=~SEX)
ctr %>% pmx_plot_eta_box(strat.facet =~SEX)

Appendix A

Monolix requirements

ggPMX is compatible with Monolix versions 2016 and later. In order to be able to produce all available diagnostic plots, the following tasks should be executed in Monolix during the model fitting procedure:

In addition, make sure to export charts data (In Monolix 2018: Settings -> Preferences -> Export -> switch on the Export charts data button). Select at least the following plots to be displayed and saved: individual fits and scatter plot of the residuals.

Plots table

The main target of ggPMX is to create a report containing the following plots (see abbreviation list below):

out <- rbind(
  c("Scatter plot of NPDE vs population predictions", "SCATTER", "npde_pred"),
  c("Scatter plot of NPDE vs time", "SCATTER", "npde_time"),
  c("Scatter plot of IWRES vs time", "SCATTER", "iwres_time"),
  c("Scatter plot of observations vs population predictions", "SCATTER", "dv_pred"),
  c("Scatter plot of observations vs individual predictions", "SCATTER", "dv_ipred"),
  c("Scatter plot of absolute value of IWRES vs individual predictions", "SCATTER", "abs_iwres_ipred"),
  c("Scatter plot of IWRES vs individual predictions", "SCATTER", "iwres_ipred"),
  c("Plots of observations and model predictions per individual", "IND", "individual"),
  c("Histogram of EBE", "DIS", "ebe_hist"),
  c("Boxplot of EBE", "DIS", "ebe_box"),
  c("Distribution and quantile-quantile plot of IWRES", "QQ", "qq_iwres"),
  c("Distribution and correlation structure of RE (`ETA`)", "ETA_PAIRS", "eta_matrix"),
  c("Relationships between RE and categorical covariates", "ETA_COV", "eta_cats"),
  c("Relationships between RE and continuous covariates", "ETA_COV", "eta_conts"),
  c("Visual predictive check (VPC)", "VPC", "vpc")

colnames(out) <- c("Plot Name", "ggPMX type", "ggPMX name")
xt <- xtable(out, label = "tab:plots_list", caption = "List of all diagnostic plots")
print(xt, comment = F)


ggPMX main functions

ggPMX implements few functions to generate and manipulate diagnostic plots. (Should we list pmx and pmx_mlx separately and say the differences? Or it's maybe clear from the previous sections.)

out <- rbind(
  c("1", "pmx, or pmx_mlx", "Creates a controller"),
  c("2", "plot_names or plots", "Lists controller plots"),
  c("3", "get_data", "Lists controller data"),
  c("4", "get_plot", "Prints a plot"),
  c("5", "set_plot", "Creates a new plot"),
  c("6", "pmx_update", "Updates an existing plot"),
  c("7", "pmx_filter", "Filters globally the data of the current session"),
  c("8", "pmx_copy", "Returns a deep copy of the controller")

colnames(out) <- c(" ", "Function name", "Description")

xt <- xtable(out, label = "tab:func_list", caption = "List of all `ggPMX` functions")
print(xt, comment = F)

(Apparently, it's not the full list. Add all functions.) The design of the package is around the central object: the controller. It can introspected or piped using the %>% operand.

Note that:

The controller is an R6 object, it behaves like a reference object. Some functions (methods) can have a side effect on the controller and modify it internally. Technically speaking we talk about chaining not piping here. However, using pmx_copy user can work on a copy of the controller.

ggPMX graphical parameters

Graphical parameters in ggPMX are set internally using the pmx_gpar function. A number of graphical parameters can be set for the different plot types.


More information can be found in the help document ?pmx_gpar and in the examples that follow.

Pre-defined configurations

For the moment we are mainly using standing configuration. In the next release user can specfiy configuration either by cretaing a custom yaml file or an R configuration object. Also ggPMX will create many helper functions to manipulate the configuration objects.


Default call

The shrinkage is a computation within controller data. In general it is used to annotate the plots. Although one can get it independently from any plot using pmx_comp_shrink. It is part of the pmx_compt_xx layer( In the future we will add , pmx_comp_cor , pmx_comp_summary,..)

Here the basic call:

ctr %>% pmx_comp_shrink

We get the shrinkage for each effect (SHRINK column).

The same values can be shown on distribution plot , for example :

ctr %>% pmx_plot_eta_box

or :

ctr %>% pmx_plot_eta_hist

You can add or remove shrinkage annotation using is.shrink argument ( TRUE by default) :

ctr %>%   pmx_plot_eta_box( is.shrink = FALSE) 

Var function

You can compute shrinkage by applying either standard deviation sd or variance var :

ctr %>% pmx_comp_shrink(  fun = "var")
ctr %>% pmx_plot_eta_box( shrink=list(fun = "var"))

Shrinkage and stratification

Shrinkage can be applied after stratification :

ctr %>% pmx_comp_shrink(strat.facet = ~SEX)

or by grouping like :

ctr %>% pmx_comp_shrink(strat.color = "SEX")

We can

ctr %>% pmx_plot_eta_hist(is.shrink = TRUE, strat.facet = ~SEX,


ctr %>% pmx_plot_eta_box(is.shrink = TRUE, strat.facet = "SEX",

ggPMXdevelopment/ggPMX documentation built on Dec. 31, 2019, 9:22 a.m.