Univariate, Bivariate and Trivariate Entropies
In netropy: Statistical Entropy Analysis of Network Data

knitr::opts_chunk$set(out.width = "100%",
  cache = FALSE
)

The univariate entropy for discrete variable $X$ with $r$ outcomes is defined by $$H(X) = \sum_x p(x) \log_2\frac{1}{p(x)} $$ with which we can check for redundancy and uniformity: a discrete random variable with minimal zero entropy has no uncertainty and is always equal to the same single outcome. Thus, it is a constant that contributes nothing to further analysis and can be omitted. Maximum entropy is $\log_2r$ and it corresponds to a uniform probability distribution over the outcomes.

The bivariate entropy for discrete variable $X$ and $Y$ is defined by $$H(X,Y) = \sum_x \sum_y p(x,y) \log_2\frac{1}{p(x,y)}$$ with which we can check for redundancy, functional relationships and stochastic independence between pairs of variables. It is bounded according to $$H(X) \leq H(X,Y) \leq H(X)+H(Y)$$ where we have

equality to the left iff there is a functional relationship $Y = f(X)$ such that each unique outcome of $X$ yields a unique outcome of $Y$
equality to the right iff $X$ and $Y$ are stochastically independent $X\bot Y$ such that the probability of any bivariate outcome is the product of the probabilities of the univariate outcomes.

Note that when the bivariate entropy of two variables is equal to the univariate entropy of either one alone, then one of these variables should be omitted as they are redundant providing no additional information.

These results on bivariate entropies are directly linked to joint entropies and association graphs.

Similarly, trivariate entropies (and higher order entropies) allows us to check for functional relationships and stochastic independence between three (or more) variables. The trivariate entropy of three variables $X$, $Y$ and $Z$ is defined by $$H(X,Y,Z) = \sum_x \sum_y \sum_z p(x,y,z) \log_2\frac{1}{p(x,y,z)}$$

and bounded by $$ H(X,Y) \leq H(X,Y,Z) \leq H(X,Z) + H(Y,Z) - H(Z). $$

The results on bivariate and trivariate entropies are directly linked to prediction power and expected conditional entropies.

Examples of computing univariate, bivariate and trivariate entropies are given in the following.

Example: univariate and bivariate entropies

library(netropy)

We create a dataframe dyad.var consisting of dyad variables as described and created in variable domains and data editing. Similar analyses can be perfomed on observed and/or transformed dataframes with vertex or triad variables.

data(lawdata) 
adj.advice <- lawdata[[1]]
adj.friend <- lawdata[[2]]
adj.cowork <-lawdata[[3]]
df.att <- lawdata[[4]]

att.var <-
  data.frame(
    status   = df.att$status-1,
    gender   = df.att$gender,
    office   = df.att$office-1,
    years    = ifelse(df.att$years<=3,0,
                      ifelse(df.att$years<=13,1,2)),
    age      = ifelse(df.att$age<=35,0,
                      ifelse(df.att$age<=45,1,2)),
    practice = df.att$practice,
    lawschool= df.att$lawschool-1
    )
dyad.status    <- get_dyad_var(att.var$status, type = 'att')
dyad.gender    <- get_dyad_var(att.var$gender, type = 'att')
dyad.office    <- get_dyad_var(att.var$office, type = 'att')
dyad.years     <- get_dyad_var(att.var$years, type = 'att')
dyad.age       <- get_dyad_var(att.var$age, type = 'att')
dyad.practice  <- get_dyad_var(att.var$practice, type = 'att')
dyad.lawschool <- get_dyad_var(att.var$lawschool, type = 'att')
dyad.cwk    <- get_dyad_var(adj.cowork, type = 'tie')
dyad.adv    <- get_dyad_var(adj.advice, type = 'tie')
dyad.frn    <- get_dyad_var(adj.friend, type = 'tie')
dyad.var <-
  data.frame(cbind(status    = dyad.status$var,
                  gender    = dyad.gender$var,
                  office    = dyad.office$var,
                  years     = dyad.years$var,
                  age       = dyad.age$var,
                  practice  = dyad.practice$var,
                  lawschool = dyad.lawschool$var,
                  cowork    = dyad.cwk$var,
                  advice    = dyad.adv$var,
                  friend    = dyad.frn$var)
                  )

head(dyad.var)

The function entropy_bivar() computes the bivariate entropies of all pairs of variables in the dataframe. The output is given as an upper triangular matrix with cells giving the bivariate entropies of row and column variables. The diagonal thus gives the univariate entropies for each variable in the dataframe:

entropy_bivar(dyad.var)

Example: redundant variables

Bivariate entropies can be used to detect redundant variables that should be omitted from the dataframe for further analysis. When calculating bivariate entropies, one can check of whether the diagonal values are equal to any of the other values in the rows an columns. As seen above, the dataframe dyad.var has no redundant variables. This can also be checked using the function redundancy() which yields a binary matrix as output indicating which row and column variables are hold the same information:

redundancy(dyad.var)

To illustrate an example with redundancy, we use the dataframe att.var with node attributes as described and created in variable domains and data editing. Note however that we now keep the variable senior in this dataframe:

att.var <-
  data.frame(
    senior   = df.att$senior,
    status   = df.att$status-1,
    gender   = df.att$gender,
    office   = df.att$office-1,
    years    = ifelse(df.att$years<=3,0,
                      ifelse(df.att$years<=13,1,2)),
    age      = ifelse(df.att$age<=35,0,
                      ifelse(df.att$age<=45,1,2)),
    practice = df.att$practice,
    lawschool= df.att$lawschool-1
    )

head(att.var)

Checking redundancy on this dataframe yields the following output:

redundancy(att.var)

As seen, senior has been flagged as a redundant variable which is not surprising since it only consists of unique values. This redudancy can also be noted by computing the bivariate entropies and noting that the univariate entropy for this variable is equal to the bivariate entropies of pairs including this variable:

entropy_bivar(att.var)

Example: trivariate entropies

Trivariate entropies can be computed using the function entropy_trivar() which returns a dataframe with the first three columns representing possible triples of variables V1,V2, and V3 from the dataframe in question, and their entropies H(V1,V2,V3) as the fourth column. We illustrated this on the dataframe dyad.var:

entropy_trivar(dyad.var)

References

Frank, O., & Shafie, T. (2016). Multivariate entropy analysis of network data. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 129(1), 45-63. link

Nowicki, K., Shafie, T., & Frank, O. (Forthcoming 2022). Statistical Entropy Analysis of Network Data.