Description Usage Arguments Format References Examples
Tools for analysis of experiments that use interleaved search results wherein users receive results from multiple sets of retrieval functions.
interleaved_data
is a fake dataset used for testing and examples;
refer to Format section below
interleaved_data_a
is a fake dataset used for testing and examples;
"A" is preferred over "B"
interleaved_data_b
is a fake dataset used for testing and examples;
"B" is preferred over "A"
interleaved_preference
returns a test statistic summarizing the
interleaving experiment; a positive value indicates that A is better
than B, a negative value indicates that B is better than A
interleaved_bootstraps
returns a bootstrapped sample of preference
statistics computed by resampling sessions with replacements
interleaved_confint
returns a list
with elements "point.est",
"lower", and "upper" (uses interleaved_bootstraps
internally)
interleaved_sample_size
estimates the sample size required to detect
a particular effect size with a specified power and significance level
1 2 3 4 5 6 7 8 9 10 11 | interleaved_bootstraps(sessions, clicks, bootstraps = 1000L)
interleaved_confint(sessions, clicks, bootstraps = 1000L, confidence = 0.95)
interleaved_preference(sessions, clicks)
interleaved_data
interleaved_data_a
interleaved_data_b
|
sessions |
vector of session IDs used to group |
clicks |
vector that shows which ranking function the clicked search result came from ("A" or "B") |
bootstraps |
number of times to sample unique sessions (with replacement); 1000 by default |
confidence |
level; 0.95 by default |
interleaved_data*
are data.frame
-s of generated search sessions with
the following columns:
10-character alphanumeric ID; for grouping events
when the event occurred; uses POSIXct format
"serp" or "click"
position ("ranking") of the clicked search result
"A" or "B"
Users in interleaved_data
have no preference, users in
interleaved_data_a
have preference for ranking function "A", and users
in interleaved_data_b
have preference for ranking function "B".
An object of class data.frame
with 5396 rows and 5 columns.
An object of class data.frame
with 5552 rows and 5 columns.
Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012). Large-scale validation and analysis of interleaved search evaluation. ACM Transactions on Information Systems, 30(1), 1-41. doi:10.1145/2094072.2094078
Radlinski, F. and Craswell, N. (2013). Optimized interleaving for online retrieval evaluation. # nolint ACM International Conference on Web Search and Data Mining (WSDM). doi:10.1145/2433396.2433429
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | data("interleaved_data", package = "wmf")
x <- interleaved_data[interleaved_data$event == "click", ]
x <- x[order(x$session_id, x$timestamp), ]
data("interleaved_data_a", package = "wmf")
y <- interleaved_data_a[interleaved_data_a$event == "click", ]
y <- y[order(y$session_id, y$timestamp), ]
data("interleaved_data_b", package = "wmf")
z <- interleaved_data_b[interleaved_data_b$event == "click", ]
z <- z[order(z$session_id, z$timestamp), ]
# Bootstrapped preference statistics:
## Data without a clear preference:
b <- interleaved_bootstraps(x$session_id, x$ranking_function)
hist(b)
## Data where A is preferred over B:
b <- interleaved_bootstraps(y$session_id, y$ranking_function)
hist(b)
## Data where B is preferred over A:
b <- interleaved_bootstraps(z$session_id, z$ranking_function)
hist(b)
# Preference statistic confidence intervals:
## Data without a clear preference:
interleaved_confint(x$session_id, x$ranking_function)
## Data where A is preferred over B:
interleaved_confint(y$session_id, y$ranking_function)
## Data where B is preferred over A:
interleaved_confint(z$session_id, z$ranking_function)
# Preference statistic calculation:
## Data without a clear preference:
interleaved_preference(x$session_id, x$ranking_function)
## Data where A is preferred over B:
interleaved_preference(y$session_id, y$ranking_function)
## Data where B is preferred over A:
interleaved_preference(z$session_id, z$ranking_function)
|
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.