Intro Examples to grizbayr

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library(grizbayr)
library(dplyr)

About the Package

Bayesian Inference is a method of statistical inference that can be used in the analysis of observed data from marketing tests. Bayesian updates start with a prior distribution (prior probable information about the environment) and a likelihood function (an expected distribution from which the samples are drawn). Then, given some observed data, the prior can be multiplied by the likelihood of the data to produce a posterior distribution of probabilities. At the core of all of this is Bayes' Rule.

$$ P(A\ |\ Data) \sim P(Data\ |\ A) \cdot P(A)$$ This package is intended to abstract the math of the conjugate prior update rules to provide 3 pieces of information for a user:

  1. Win Probability (overall and vs baseline)
  2. Value Remaining
  3. Lift vs. Control

Usage

Select which piece of information you would like to calculate.

| Metric | Function Call | |------------------------------|-----------------------------------| | All Below Metrics | calculate_all_metrics() | | Win Probability | estimate_win_prob() | | Value Remaining | estimate_value_remaining() | | Lift vs. Control | estimate_lift_vs_baseline() | | Win Probability vs. Baseline | estimate_win_prob_vs_baseline() |

If you would like to calculate all the metrics then use calculate_all_metrics(). This is a slightly more efficient implementation since it only needs to sample from the posterior once for all 4 calculations instead of once for each metric.

Create an Input Dataframe or Tibble

All of these functions require a very specific tibble format. However, the same tibble can be used in all metric calculations. A tibble is used here because it has the additional check that all column lengths are the same. A tibble of this format can also conveniently be created using dplyr's group_by() %>% summarise() sequence of functions.

The columns in the following table are required if there is an X in the box for the distribution. (Int columns can also be dbl due to R coercian)

| Distribution Type | option_name (char) | sum_impressions (int) | sum_clicks (int) | sum_sessions (int) | sum_conversions (dbl) | sum_revenue (dbl) | sum_cost (dbl) | sum_conversions_2 (dbl) | sum_revenue_2 (dbl) | sum_duration (dbl) | sum_page_views (int) | |---------------------------|:------------------:|:---------------------:|:----------------:|:------------------:|:---------------------:|:-----------------:|:--------------:|:-----------------------:|:-------------------:|:------------------:|:--------------------:| | Conversion Rate | X | | X | | X | | | | | | | | Response Rate | X | | | X | X | | | | | | | | Click Through Rate (CTR) | X | X | X | | | | | | | | | | Revenue Per Session | X | | | X | X | X | | | | | | | Multi Revenue Per Session | X | | | X | X | X | | X | X | | | | Cost Per Activation (CPA) | X | | X | | X | | X | | | | | | Total CM | X | X | X | | X | X | X | | | | | | CM Per Click | X | | X | | X | X | X | | | | | | Cost Per Click (CPC) | X | | X | | | | X | | | | | | Session Duration | X | | | X | | | | | | X | | | Page Views Per Session | X | | | X | | | | | | | X |

Example:

We will use the Conversion Rate distribution for this example so we need the columns option_name, sum_clicks, and sum_conversions.

raw_data_long_format <- tibble::tribble(
   ~option_name, ~clicks, ~conversions,
            "A",       6,           3,
            "A",       1,           0,
            "B",       2,           1,
            "A",       2,           0,
            "A",       1,           0,
            "B",       5,           2,
            "A",       1,           0,
            "B",       1,           1,
            "B",       1,           0,
            "A",       3,           1,
            "B",       1,           0,
            "A",       1,           1
)

raw_data_long_format %>% 
  dplyr::group_by(option_name) %>% 
  dplyr::summarise(sum_clicks = sum(clicks), 
                   sum_conversions = sum(conversions))

This input dataframe can also be created manually if the aggregations are already done in an external program.

# Since this is a stochastic process with a random number generator,
# it is worth setting the seed to get consistent results.
set.seed(1776)

input_df <- tibble::tibble(
  option_name = c("A", "B", "C"),
  sum_clicks = c(1000, 1000, 1000),
  sum_conversions = c(100, 120, 110)
)
input_df

One note: clicks or sessions must be greater than or equal to the number of conversions (this is a rate bound between 0 and 1).

input_df is used in the following examples.

Estimate All Metrics

This function wraps all the below functions into one call.

estimate_all_values(input_df, distribution = "conversion_rate", wrt_option_lift = "A")

Win Probability

This produces a tibble with all the option names, the win_prob_raw so this can be used as a double, and a cleaned string win_prob where the decimal is represented as a percent.

estimate_win_prob(input_df, distribution = "conversion_rate")

Value Remaining (Loss)

Value Remaining is a measure of loss. If B is selected as the current best option, we can estimate with 95% confidence (default), that an alternative option is not more than X% worse than the current expected best option.

estimate_value_remaining(input_df, distribution = "conversion_rate")

This number can also be framed in absolute dollar terms (or percentage points in the case of a rate metric).

estimate_value_remaining(input_df, distribution = "conversion_rate", metric = "absolute")

Estimate Lift

The metric argument defaults to lift which produces a percent lift vs the baseline. Sometimes we may want to understand this lift in absolute terms (especially when samples from the posteriors could be negative, such as Contribution Margin (CM).)

estimate_lift_vs_baseline(input_df, distribution = "conversion_rate", wrt_option = "A")
estimate_lift_vs_baseline(input_df, distribution = "conversion_rate", wrt_option = "A", metric = "absolute")

Win Probability vs. Baseline

This function is used to compare an individual option to the best option as opposed to the win probability of each option overall.

estimate_win_prob_vs_baseline(input_df, distribution = "conversion_rate", wrt_option = "A")

Sample From the Posterior

Samples can be directly collected from the posterior with the following function.

sample_from_posterior(input_df, distribution = "conversion_rate")

Alternate Distribution Type (Rev Per Session)

(input_df_rps <- tibble::tibble(
   option_name = c("A", "B", "C"),
   sum_sessions = c(1000, 1000, 1000),
   sum_conversions = c(100, 120, 110),
   sum_revenue = c(900, 1200, 1150)
))

estimate_all_values(input_df_rps, distribution = "rev_per_session", wrt_option_lift = "A")

Valid Posteriors

You may want to pass alternate priors to a distribution. Only do this if you are making an informed decision.

Beta - alpha0, beta0
Gamma - k0, theta0 (k01, theta01 if alternate Gamma priors are required)
Dirichlet - alpha_00 (none), alpha_01 (first conversion type), alpha_02 (alternate conversion type)
# You can also pass priors for just the Beta distribution and not the Gamma distribution.
new_priors <- list(alpha0 = 2, beta0 = 10, k0 = 3, theta0 = 10000)
estimate_all_values(input_df_rps, distribution = "rev_per_session", wrt_option_lift = "A", priors = new_priors)

Looping Over All Distributions

You may want to evaluate the results of a test in multiple different distributions.

(input_df_all <- tibble::tibble(
   option_name = c("A", "B", "C"),
   sum_impressions = c(10000, 9000, 11000),
   sum_sessions = c(1000, 1000, 1000),
   sum_conversions = c(100, 120, 110),
   sum_revenue = c(900, 1200, 1150),
   sum_cost = c(10, 50, 30),
   sum_conversions_2 = c(10, 8, 20),
   sum_revenue_2 = c(10, 16, 15)
) %>% 
  dplyr::mutate(sum_clicks = sum_sessions)) # Clicks are the same as Sessions

distributions <- c("conversion_rate", "response_rate", "ctr", "rev_per_session", "multi_rev_per_session", "cpa", "total_cm", "cm_per_click", "cpc")

# Purrr map allows us to apply a function to each element of a list. (Similar to a for loop)
purrr::map(distributions,
           ~ estimate_all_values(input_df_all,
                                 distribution = .x,
                                 wrt_option_lift = "A",
                                 metric = "absolute")
)


Try the grizbayr package in your browser

Any scripts or data that you put into this service are public.

grizbayr documentation built on Oct. 9, 2023, 5:10 p.m.