interleaved: Interleaved search results

In wikimedia/wikimedia-discovery-ortiz: Speedy Dwelltime and Search Preference Computations

Description

Tools for analysis of experiments that use interleaved search results wherein users receive results from multiple sets of retrieval functions.

• interleaved_data is a fake dataset used for testing and examples; refer to Format section below

• interleaved_data_a is a fake dataset used for testing and examples; "A" is preferred over "B"

• interleaved_data_b is a fake dataset used for testing and examples; "B" is preferred over "A"

• interleaved_preference returns a test statistic summarizing the interleaving experiment; a positive value indicates that A is better than B, a negative value indicates that B is better than A

• interleaved_bootstraps returns a bootstrapped sample of preference statistics computed by resampling sessions with replacements

• interleaved_confint returns a list with elements "point.est", "lower", and "upper" (uses interleaved_bootstraps internally)

• interleaved_sample_size estimates the sample size required to detect a particular effect size with a specified power and significance level

Usage

interleaved_bootstraps(sessions, clicks, bootstraps = 1000L)

interleaved_confint(sessions, clicks, bootstraps = 1000L, confidence = 0.95)

interleaved_preference(sessions, clicks)

interleaved_data

interleaved_data_a

interleaved_data_b

Arguments

sessions	vector of session IDs used to group positions and ranking_functions
clicks	vector that shows which ranking function the clicked search result came from ("A" or "B")
bootstraps	number of times to sample unique sessions (with replacement); 1000 by default
confidence	level; 0.95 by default

Format

interleaved_data* are data.frame-s of generated search sessions with the following columns:

session_id

10-character alphanumeric ID; for grouping events

timestamp

when the event occurred; uses POSIXct format

event

"serp" or "click"

position

position ("ranking") of the clicked search result

ranking_function

"A" or "B"

Users in interleaved_data have no preference, users in interleaved_data_a have preference for ranking function "A", and users in interleaved_data_b have preference for ranking function "B".

An object of class data.frame with 5396 rows and 5 columns.

An object of class data.frame with 5552 rows and 5 columns.

References

• Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012). Large-scale validation and analysis of interleaved search evaluation. ACM Transactions on Information Systems, 30(1), 1-41. doi:10.1145/2094072.2094078

• Radlinski, F. and Craswell, N. (2013). Optimized interleaving for online retrieval evaluation. # nolint ACM International Conference on Web Search and Data Mining (WSDM). doi:10.1145/2433396.2433429

Examples

data("interleaved_data", package = "wmf")
x <- interleaved_data[interleaved_data$event == "click", ]
x <- x[order(x$session_id, x$timestamp), ]
data("interleaved_data_a", package = "wmf")
y <- interleaved_data_a[interleaved_data_a$event == "click", ]
y <- y[order(y$session_id, y$timestamp), ]
data("interleaved_data_b", package = "wmf")
z <- interleaved_data_b[interleaved_data_b$event == "click", ]
z <- z[order(z$session_id, z$timestamp), ]

# Bootstrapped preference statistics:

## Data without a clear preference:
b <- interleaved_bootstraps(x$session_id, x$ranking_function)
hist(b)

## Data where A is preferred over B:
b <- interleaved_bootstraps(y$session_id, y$ranking_function)
hist(b)

## Data where B is preferred over A:
b <- interleaved_bootstraps(z$session_id, z$ranking_function)
hist(b)

# Preference statistic confidence intervals:

## Data without a clear preference:
interleaved_confint(x$session_id, x$ranking_function)

## Data where A is preferred over B:
interleaved_confint(y$session_id, y$ranking_function)

## Data where B is preferred over A:
interleaved_confint(z$session_id, z$ranking_function)

# Preference statistic calculation:

## Data without a clear preference:
interleaved_preference(x$session_id, x$ranking_function)

## Data where A is preferred over B:
interleaved_preference(y$session_id, y$ranking_function)

## Data where B is preferred over A:
interleaved_preference(z$session_id, z$ranking_function)

wikimedia/wikimedia-discovery-ortiz documentation built on Aug. 23, 2020, 9:46 a.m.