dstudy: D-study (Decision)

In julian-urbano/gt4ireval: Generalizability Theory for Information Retrieval Evaluation

Description

dstudy runs a D-study from the results of a gstudy and computes, for a certain number of queries, the expected generalizability coefficient Erho2 and index of dependability Phi, possibly with confidence intervals. Alternatively, it can estimate the number of queries needed to achieve a certain level of stability, also with confidence intervals.

Usage

dstudy(gdata, queries = gdata$n.q, stability = 0.95, alpha = 0.025)

Arguments

gdata	The result of running a gstudy with existing data.
queries	A vector with different query set sizes for which to estimate Erho2 and Phi. Defaults to the number of queries used to compute gdata.
stability	A vector with target Erho2 and Phi values to estimate required query set sizes.
alpha	A vector of confidence levels to compute intervals for Erho2, Phi and query set sizes. This is the probability on each side of the interval, so for a 90% confidence interval one must set alpha to 0.05.

Value

An object of class dstudy, with the following components:

Erho2, Erho2.lwr, Erho2.upr	Expected generalizability coefficient, and lower and upper limits of the intervals around it.
Phi, Phi.lwr, Phi.upr	Expected index of dependability, and lower and upper limits of the intervals around it.
n.q_Erho2, n.q_Erho2.lwr, n.q_Erho2.upr	Expected number of queries to achieve the generalizability coefficient, and lower and upper limits of the intervals around it.
n.q_Phi, n.q_Phi.lwr, n.q_Phi.upr	Expected number of queries to achieve the index of dependability, and lower and upper limits of the intervals around it.
call	A list with the gstudy used in this D-study, the target number of queries, target level of stability and alpha level for the confidence intervals.

Author(s)

Julián Urbano

References

R.L. Brennan (2001). Generalizability Theory. Springer.

L.S. Feldt (1965). The Approximate Sampling Distribution of Kuder-Richardson Reliability Coefficient Twenty. Psychometrika, 30(3):357–370.

C. Arteaga, S. Jeyaratnam, and G. A. Franklin (1982). Confidence Intervals for Proportions of Total Variance in the Two-Way Cross Component of Variance Model. Communications in Statistics: Theory and Methods, 11(15):1643–1658.

J. Urbano, M. Marrero and D. Martín (2013). On the Measurement of Test Collection Reliability. ACM SIGIR, pp. 393-402.

See Also

gstudy, gt2tau

Examples

g <- gstudy(adhoc3)
dstudy(g)

# estimate stability at various query set sizes
dstudy(g, queries = seq(50, 200, 10))
# estimate required query set sizes for various stability levels
dstudy(g, stability = seq(0.8, 0.95, 0.01))
# compute both 95% and 99% confidence intervals
dstudy(g, stability = 0.9, alpha = c(0.05, 0.01) / 2)
# compute 1-tailed 95% confidence intervals
dstudy(g, alpha = 0.05)

julian-urbano/gt4ireval documentation built on May 20, 2019, 4:21 a.m.