success_vs_bench: Compare a success rate to a benchmark
In darrellpenta/uxstats: Common data analysis tools for UX researchers

Description Usage Arguments Details Value See Also Examples

View source: R/success_vs_bench-function.R

success_vs_bench() tests an observed success rate against a given benchmark. Following Sauro and Lewis (2012), it takes the sample size into account in providing estimates.

success_vs_bench() and completion_vs_bench() are synonyms.

success_vs_bench(.x, ...)

completion_vs_bench(.x, ...)

## S3 method for class 'numeric'
success_vs_bench(
  .x,
  .n = NULL,
  .p,
  ...,
  .alt = c("greater", "less", "twotailed"),
  .alpha = 0.05
)

## S3 method for class 'data.frame'
success_vs_bench(
  .x,
  .var,
  .p,
  ...,
  .alt = c("greater", "less", "twotailed"),
  .alpha = 0.05
)

`.x`	A single numeric value, a vector of values, or a long-format data frame with a named column of numeric data (1s and/or 0s) corresponding to task success outcomes. See Details.
`...`	(Optional) If `.x` is a long-format data frame, you can pass the name of one or more grouping variables as unquoted, comma-separated column names (without naming the `...` argument) to compute stats by groups.
`.n`	A single numeric value representing the total number of trials. See Details.
`.p`	The test (benchmark) proportion (must be a numeric between 0-1).
`.alt`	For test alternatives, one of `c("greater","less","twotailed")`. Defaults to "greater" for a one-sided test.
`.alpha`	(Optional) A positive number (where 0 < `.alpha` < 1) specifying the significance level to be used. Defaults to `.alpha = 0.05`. To set a different significance level, the argument must be named (i.e., `.alpha=0.001`) or else the function may yield unexpected results.
`.var`	If `.x` is a long-format data frame, the (unquoted) name of a data frame column containing task success outcomes (as 1s and 0s, corresponding to successes and failures, respectively).

success_vs_bench() returns a variety of estimates. Sauro and Lewis (2012) recommend using the mid-probability from the binomial distribution for small sample sizes (i.e., cases with fewer than 15 successes and 15 failures), and for large sample sizes, using the normal approximation to the binomial. The function also reports the best estimate success rate using the Laplace calculation.

success_vs_bench assumes that you want to test the hypothesis that the observed outcome exceeds the benchmark, and therefore, defaults to a one-tailed test. This means that setting .alpha = 0.05 (the default) produces a 90% confidence interval.

If .x is a single numeric value representing the total number of successes, .n should be a single numeric value representing the total number of users, where the value of .n >= the value of .x). e.g., .x = 23, .n = 25
If .x is a data frame, .var should be the unquoted name of the column containing the success data (as 1s and 0s).
If you're passing a data frame to .x, you can optionally pass one or more grouping variables as unquoted, comma-separated column names (without naming the ... argument) to compute stats by groups.
You can choose from among the test alternatives c("greater","less","twotailed") by providing one of the options to the .alt argument: e.g., .alt = "twotailed". Defaults to "greater" for a one-sided test.
You can modify the alpha level to adjust confidence intervals by including .alpha as a named argument and providing a numeric value: e.g., .aplha = 0.001.

Note that NAs are automatically dropped in all calculations.

A tibble with data summaries and test results

Other benchmark comparison stats: ratings_vs_bench(), time_vs_bench()

# Comparing 19 success/25 trials (users) to a 75% benchmark completion rate
success_vs_bench(19,25,0.75)

.ux_data <-
data.frame(
 "id" = rep(seq(1,10,1),2),
 "task" = c(rep(1,10),rep(2,10)),
 "complete"  = sample(0:1,20,replace=TRUE,prob = c(.3,.65))
)

success_vs_bench(.ux_data, complete, .p=0.7,task)