In google/GeoexperimentsResearch: Geo Experiments

set.seed(20170224L)
options(width=99)

Geo Experiments: an introduction

What are geo experiments?

A geo experiment is a controlled online experiment, with the aim of quantifying the effects of treatments using geographical regions as experimental units.

In its simplest form, a geographical region is divided into a Control and a Treatment region; the people in the Treatment region are exposed to a certain treatment, while for those in the Control region nothing changes. The behavior (response of some given observable variable) in both these regions is observed and measured and the effect of the treatment is estimated using a statistical model.

The Control and Treatment regions are formed by aggregating smaller geographical units (called 'geos'), usually by randomization or by a matching algorithm.

Geo experiments for estimating ad effectiveness

A typical application is to investigate the effect of increased investment in online advertising. People who reside in the Treatment region are served ads at a higher intensity (spend per time unit) whenever they do a web search using prespecified keywords. Those who reside in the Control region continue to be served ads at the usual intensity. Such an experiment typically lasts for a few weeks.

The aim of a geo experiment in the online ad context is to estimate the return (in monetary terms) on the money spent: more accurately the incremental return on ad spend (iROAS), which refers to the additional return that would have not been received without the specific additional spend in ads. The "return" is usually the aggregate sales, either online or offline.

Similarly, we can estimate the effect of decreased spending. So the market intervention here may be either "increased" spending, "decreased" spending, or in general, some spend "change". In the simple one-period geo experiments, the statistical model only needs to know which geos experienced a spend change and which did not (i.e., which geos were in the Control regions).

Overview of this vignette

This vignette shows briefly how to use the R package 'GeoexperimentsResearch,'

1. to represent information as data objects;
2. to analyze geo experiment data once an experiment has finished;
3. to run a preanalysis.

For details, refer to the package manual.

For further general information on geo experiments, see [1], [2], and [3].

Attaching the package

library(GeoexperimentsResearch)

Structure of geo experiment data

For the purpose of illustration, we assume that we have run a geo experiment and wish to estimate the incremental return on ad spend.

We need to collect three pieces of information:

• The experiment periods (date ranges of the Pretest, intervention, and cooldown periods);
• The geo assignment;
• The observational data for both the response metric and the cost metric.

Experiment periods

A geo experiment consists of several, distinct time periods: the Pretest, Intervention, and Cooldown periods. The latter two periods combined make up the `Test' period.

During the Pretest period, any ad campaigns in the Treatment and Control geos that are targeted by the experiment are in their unmodified base state. All geos operate with the same baseline level campaign settings (e.g., common bidding, keyword lists, ad targeting, etc); the difference between the Control and Treatment geos is zero in expectation.

The targeted ad campaigns are modified in the Treatment geos during the Intervention period.

Finally, these targeted ad campaigns are reset to their original state during the Cooldown period. This does not always mean their effects will cease instantly. Incremental offline sales, for example, may continue to accrue across subsequent days or even weeks. Including data from the Cooldown period in the analysis makes it possible to capture these lagged effects from the advertising change. This lagged impact may be substantial or not, depending on the advertising situation; hence it can be excluded if it is obvious in the analysis that there are no lagged effects.

By convention, we number the periods as 0 (Pretest), 1 (Intervention), 2 (Cooldown), but other numbering is allowed provided that the order of the periods is unchanged.

This information is represented by the ExperimentPeriods object class. The start dates of each period must be specified, and finally end date of the experiment. This example has only a pretest period and one intervention period:

  print(ExperimentPeriods(c("2015-01-05", "2015-02-16", "2015-03-15")))

Geo assignment

Before the experiment is started, each of the geos is assigned either to Control or to Treatment region (geo group). This mapping between a geo to the geo group is called the geo assignment.

This information is represented by the GeoAssignment object class. Example:

data(geoassignment)
head(GeoAssignment(geoassignment))

Observational data

The observational data consist of a response metric (such as sales) and cost metric (such as cost of ad clicks). These are provided broken by date and geo, including the Pretest, Intervention, and Cooldown periods. If the data is weekly data, the weekly aggregate should be associated with the same day of the month, for instance, each Sunday.

This information is represented by the GeoTimeseries object class. Example: a few rows shown from the example data set:

data(salesandcost)
head(GeoTimeseries(salesandcost, metrics=c("sales", "cost")))

Analyzing geo experiment data using the GBR and TBR methods

Reading in observational data

Load the sample data set.

data(salesandcost)

This is a plain data.frame, with the following columns:

head(salesandcost)

This data frame has a date and a geo column, and two metrics, the sales and the cost of ad clicks, numeric values that are associated to each geo and date.

Next, we convert this data frame into a GeoTimeseries object and the integrity of the time series is automatically checked. We need to specify which columns are to be treated as metrics. This helps the class methods do certain operations automatically, such as aggregation over geos and time.

obj.gts <- GeoTimeseries(salesandcost, metrics=c("sales", "cost"))

No errors occurred, so the overall structure of the data seems to be fine. The resulting object inherits from data.frame, with the same columns, augmented with some extra columns:

head(obj.gts)

The 'date' column must be in either 'Date', factor, or character format and is always coerced to Date. If the date format differs from 'yyyy-mm-dd', it is necessary to specify it as argument 'date.format'.

The column 'geo' is of type character even though some geo IDs (such as DMAs) are represented as integers. Using character format, however, the structure of GeoTimeseries is also compatible with non-integer geos such as postal codes and administrative regions without remapping them to numbers.

There is no checking of whether any of the metrics are negative.

There are some extra columns provided for convenience:

• .weekday column denotes the day of the week (1=Monday, 7=Sunday);
• .weeknum indicates the number of the week within a year;
• .weekindex indicates a unique week number.

The data frame can have any number of other columns, although the built-in methods recognize only 'date', 'geo', '.weekday', '.weeknum', and '.weekindex', and those registered as metrics.

Exploratory data analysis

To quickly investigate the distribution of the metrics across weeks, we can use the aggregate method as follows:

aggregate(obj.gts, by='.weekindex')

We can see that normally the ad campaigns were turned off, and starting from week 6, the ad spend increased until it was turned off again on week 14.

To plot the time series, use the plot method:

 plot(obj.gts)

To hide the legend, add legend=FALSE. To plot the time series on log scale, add log.scale=TRUE. For more information of the method, type ?plot.GeoTimeseries at the R prompt.

Experiment Periods

We specify the start of the Pretest period, the start of the test period, and the end of the experiment. If there is a Cooldown period after the actual market intervention, it must be included as a separate period (four dates in total).

obj.per <- ExperimentPeriods(c("2015-01-05", "2015-02-16", "2015-03-15"))
obj.per

To introduce a cooldown period, we would specify one more date.

To learn more about the function, type ?ExperimentPeriods at the R prompt.

Geo Assignment

We'll use the built-in sample geo assignment:

data(geoassignment)
head(geoassignment)

From this data frame we create a GeoAssignment object and automatically verify its integrity:

obj.ga <- GeoAssignment(geoassignment)
head(obj.ga)

Combining all information about the experiment into one object

The class GeoExperimentData combines these three pieces of information (geo time series, periods, geo assignment) into one object:

  obj <- GeoExperimentData(obj.gts,
                           periods=obj.per,
                           geo.assignment=obj.ga)
  head(obj)

The column period contains the indicator for the experiment periods: 0 = Pretest, 1 = test (Intervention). 'NA' marks a date that is outside of the designated experiment periods.

The column geo.group contains the geo group ID for each of the geos.

The column assignment is not used in this version of the R package. It is set to NA by default. It can be ignored.

Exploratory data analysis

To check how the revenue and cost metrics are distributed across periods and groups, we make use of the aggregate method again:

aggregate(obj, by=c('period', 'geo.group'))

Geo-Based Regression (GBR) Analysis

The object ('obj') that we constructed contains now all information for applying a geo experiment analysis methodology.

To perform a GBR (geo-based regression) analysis, apply method DoGBRROASAnalysis, specifying which of the metrics is the response and which represents the cost, along with the experiment periods and group numbers.

  result <- DoGBRROASAnalysis(obj, response='sales', cost='cost',
                              pretest.period=0,
                              intervention.period=1,
                              cooldown.period=NULL,
                              control.group=1,
                              treatment.group=2)
  result

Note that in this particular case, there is no cooldown.period, hence it is set to NULL. If there was one, we would specify the period number (for example, cooldown.period=2).

The resulting object (a GBRROASAnalysisFit object) contains the model fit: when printed, it shows its summary, which defaults to 90 percent credible intervals. To recalculate the interval with a different credibility level, we can specify this in the function call:

  summary(result, level=0.95, interval.type="two-sided")

To obtain the posterior probability that the true iROAS is larger than some threshold, say 3.0, we use the summary method as follows:

  summary(result, threshold=3.0)

The default threshold is 0.

Time-Based Regression (TBR) ROAS Analysis

The GeoExperimentData object can also be used for performing a TBR analysis [3], applying method DoTBRROASAnalysis, specifying which of the metrics is the response and which represents the cost, along with the experiment period and group numbers. The model ID is also required; currently the only available model is 'tbr1', as described in [3].

  obj.tbr.roas <- DoTBRROASAnalysis(obj, response='sales', cost='cost',
                                    model='tbr1',
                                    pretest.period=0,
                                    intervention.period=1,
                                    cooldown.period=NULL,
                                    control.group=1,
                                    treatment.group=2)
  obj.tbr.roas

The resulting object (a TBRROASAnalysisFit object) contains the model fit: when printed, it shows its summary, which defaults to 90 percent one-sided credible intervals. Similarly to what we did with a GBRROASAnalysisFit object we can recalculate the credible interval, and the probability of exceeding a given threshold like so:

  summary(obj.tbr.roas, level=0.95, interval.type="two-sided")

  summary(obj.tbr.roas, threshold=3.0)

The plot method shows the evolution of the iROAS estimate across the Test period:

 plot(obj.tbr.roas)

For more information on the method, type ?plot.TBRROASAnalysisFit at the R prompt.

Time-Based Regression (TBR) Causal Effect Analysis

Unlike the TBR ROAS Analysis, which estimates the ratio of the incremental response and incremental cost, the TBR Causal Effect Analysis applies only to one single variable, such as revenue.

  obj.tbr <- DoTBRAnalysis(obj, response='sales',
                           model='tbr1',
                           pretest.period=0,
                           intervention.period=1,
                           cooldown.period=NULL,
                           control.group=1,
                           treatment.group=2)

The resulting object (a TBRAnalysisFitTbr1 object) contains the model fit for each time point, which can be seen when printed. To show the summary of the effect, we use the summary method:

  summary(obj.tbr)

which defaults to the 90% one-sided interval.

The plot method illustrates the results of the analysis.

 plot(obj.tbr)

For more information on the method, type ?plot.TBRAnalysisFitTbr1 at the R prompt.

Preanalysis

Before running an experiment, we need to understand how the design parameters affect the uncertainty of the iROAS estimate. One of the most important parameters is the ad spend change, which affects the estimate uncertainty directly: doubling the ad spend halves the width of the 2-sided confidence interval (in terms of one-sided intervals, this is the distance from the lower bound and the point estimate). We refer to this confidence interval half-width by precision (which gets better as the confidence interval gets shorter).

The function DoROASPreanalysis predicts the precision of the iROAS estimate based on historical data provided. It simulates experiments (by resampling) with given period lengths and records the precision from each simulated experiment. We can then use the summary method to compute the precision given an ad spend change, or find the ad spend change associated with a given precision.

For each simulated geo experiment data set, ROAS and its precision is estimated. The process yields a distribution of the these estimates of precision. The summary method takes the empirical median as the point estimate. If the data set does not have strong seasonalities, the variation of this estimate should be fairly small.

The process runs as follows:

1. Assign geos to treatment groups.
2. Run preanalysis to predict the precision.

A randomized geo assignment

Randomized geo assignments can be done using `GeoStrata' objects. This object includes a mapping from each geo to a stratum (or block), so a stratified randomization can be performed. This can be generated automatically using the ExtractGeoStrata function:

obj.geo.strata <- ExtractGeoStrata(obj.gts, volume="sales", n.groups=2)
head(obj.geo.strata)

The argument 'volume' specifies the name of the metric that is used for stratification: the geos are sorted by their volume and divided into strata of 2 each.

To generate a randomized geo assignment, we use the `Randomize' method:

obj.geo.assignment <- Randomize(obj.geo.strata)
head(obj.geo.assignment)

Predicting the precision

We pass this object to the method DoGBRPreanalysis along with the GeoTimeseries, the length of the Pretest, Intervention, and Cooldown periods, and specify a metric:

obj.pre <-  DoROASPreanalysis(obj.gts, response="sales",
                              geos=obj.geo.assignment,
                              prop.to="sales",
                              period.lengths=c(42, 21, 7))

The resulting object 'obj.pre' is of class ROASPreanalysisFit, which only contains the raw simulated numbers. To compute the required spend for precision +/- 1.0, we call the summary method:

  results <- summary(obj.pre,
               level=0.90,
               type="one-sided",
               precision=1.0)
  print(results)

This function takes the median of the simulated precisions as the point estimate.

For convenience, applying the 'print' method on the "GBRPreanalysisFit" prints the default summary (one-sided confidence interval at level 0.90, precision 1.0):

  print(obj.pre)

The function can be also used to predict the precision given a (total) spend change over the test period:

  results2 <- summary(obj.pre,
                level=0.90,
                type="one-sided",
                cost=10000)
  print(results2)

The results apply to the given geo assignment only; for a different geo assignment, the results are likely to be different.

References

[1] Kerman, J., Vaver, J. and Koehler, J. (2011) Estimating causal effects using geo experiments

[2] Vaver, J. and Koehler, J. (2011) Measuring Ad Effectiveness Using Geo Experiments

[3] Kerman, J. and Wang, P., and Vaver, J. (2017) Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework

Disclaimer

This software is not an official Google product. For research purposes only. Copyright 2017 Google, Inc.

google/GeoexperimentsResearch documentation built on May 17, 2019, 7:42 a.m.