Description Usage Arguments Value Author(s) References See Also Examples
Returns an array of ordinal dominance statistics based on the input of two 1-column matrices as an alternative to independent or paired group mean comparisons (especially for Cliff's delta statistics).
1 2 3 |
x |
A 1-column matrix with optional column name containing all n_x values or scores of group X or 1 (e.g. control or pretest group.), e.g. declared in R as |
y |
A 1-column matrix with optional column name containing all n_y values of group Y or 2 (e.g. experimental or post-test group). For paired comparisons (e.g. pre-post), n_x = n_y is required. If y is a vector, a default column name is assigned. |
alpha |
Significance or α-level used for the calculation of the confidence intervals. Default value is α = .05 or 5 Percent, giving a 95 Percent CI. For multiple dominance comparisons, a Bonferroni procedure may be implemented: Cliff (1996, p.150) suggested dividing α by the number of possible comparisons, i.e. alpha / (.5*k(k-1)) for comparisons beteen k data sets. |
paired |
By default, independence of the two groups or data sets is assumed. If the number of cases in x and y are equal and paired (e.g. pre-post) comparisons, this should be set to TRUE to return the full array of within, between, combined and metric delta statistics. |
outputfile |
If a a detailed report of the ordinal dominance analysis is wanted, a filename should be given here. The report as standard text file is written to the current working directory. |
studdist |
By default, it is assumed that small samples are being examined. In this case, z-values based on Student's t-distribution are used for estimating upper and lower limits of the confidence intervals (CI) as well as z-probabilities. If larger sample sizes are used, these values approximate estimates based on normally distributed z-values. In this case or if comparing with estimates calculated with orddom versions <1.5 (where z-values based on the Standard Normal Distributions were used), this parameter may be set to FALSE. |
symmetric |
By default, asymmetric confidence intervals (CI) are being calculated to compensate for positive correlations between the samples as generally recommended by the literature on the delta statistics. To increase power in certain cases, however - e.g. in small paired samples (cf. Cliff 1996, p. 165) or fur purposes of evaluating the CIs of a combined delta estimate in the paired case - symmetric CIs may also be obtained by setting this argument to TRUE. |
onetailed |
By default, calculation of p values and confidence intervals (CI) assumes two-sided testing against the null hypothesis. Set to TRUE if the alternative hypothesis targets at one-tailed testing. |
t.welch |
By default, for calculation of the t-test scores and metric p and df values, the Welch approximation is used. If set to FALSE, equal variances are assumed for groups X and Y and a pooled variance is being calculated. |
x.name |
By default, the label of group x (i.e. 1st or control or pretest group) is taken from the column name of the x input matrix. This argument allows for assigning an alternative label. |
y.name |
This argument allows for assigning an alternative label for the y input matrix or group y (i.e. 2nd or experimental or posttest group). |
description |
This argument allows for assigning a string (as title or description) for the ordinal comparison outputs. |
INDEPENDENT GROUPS (paired argument set to FALSE)
In the case of independent groups or data sets X and Y (e.g. comparison group X vs. treatment group Y), a 2-column-matrix containing 29 rows with values is returned.
The ordinal statistics can be retrieved from the first column (named "ordinal") while the second column (named "metric") contains metric comparison data where appropriate.
[1 or ["var1_X", col#] |
Label assigned to group x (x.name or column name of the x input matrix) or a default "1st var (x)". |
[2 or ["var2_Y", col#] |
Label assigned to group x (x.name or column name of the x input matrix) or a default "2nd var (y)". |
[3 or ["type_title", col#] |
Column 1: Returns type of the comparison, in this case "indep". |
[4 or ["n in X", col#] |
Number of cases in x (i.e. group X sample size). |
[5 or ["n in Y", col#] |
Number of cases in y (i.e. group Y sample size). |
[6 or ["N #Y>X", col#] |
Number of occurences of an observation from group y having a higher value than an observation from group x when comparing all x scores with all y scores: N_{\#Y>X}=\#(y_i>x_j) , where \# denotes "the number of times" whilst comparing each i=1, 2, 3, … n_y score in sample Y with each j=1, 2, 3, … n_x score in sample X (resulting in (n_x)(n_y) comparisons). |
[7 or ["N #Y=X", col#] |
Number of occurences of an observation from group y having the same value as an observation from group x: N_{\#Y=X}=\#(y_i=x_j). |
[8 or ["N #Y<X", col#] |
Number of occurences of an observation from group y having a smaller value than an observation from group x: N_{\#Y<X}=\#(y_i<x_j). |
[9 or ["PS X>Y", col#] |
Common Language CL effect size or Probability of Superiority (PS) of X over Y, see below. |
[10 or ["PS Y>X", col#] |
Column 1: Discrete case Common Language CL effect size or Probability of Superiority (PS) of Y over X,PS(Y>X)=\#(y_i>x_j)/(n_y n_x) (cf. Grissom, 1994,Grissom & Kim, 2005,McGraw & Wong, 1992). This effect size reflects the probability that a subject or case randomly chosen from group Y has a higher score than than a randomly chosen subject or case from group X (cf. Acion et al., 2006). |
[11 or ["A X>Y", col#] |
Vargha and Delaney's A as stochastic superiority of X over Y, calculated as A(X>Y) = PS(X>Y)+.5 PS(X=Y) (cf. Vargha & Delaney, 1998, 2000,Delaney & Vargha, 2002). This modified probability of superiority effect size has also been called area under the the receiver operating characteristic curve or AUC by Kraemer and Kupfer (2006). If one sampled one single case or subject from group Y and one from group X, respectively, A or AUC is the probability that the sample taken from group Y has a higher score or value than the one sampled from X (given the toss of a coin to break any ties). See also codedmes of this package. |
[12 or ["A Y>X", col#] |
Vargha and Delaney's A as stochastic superiority of Y over X. |
[13 or ["delta", col#] |
For column 1 ("ordinal"): Cliff's delta for independent groups (Cliff, 1996,Long et al., 2003): d=SUM(SUM(d_ij))/(n_x*n_y) where d_ij=sign(y_i-x_j) across all score comparisons. Termed success rate difference (SRD) effect size by Kraemer and Kupfer, delta denotes the difference between the probability that a randomly chosen Y case or subject (or patient) has a higher score than a randomly chosen case or subject from group X and the probability for the opposite. |
[14 or ["1-alpha", col#] |
Significance or α-level for CI estimation, given as percentage between 0 and 100. |
[15 or ["CI low", col#] |
Unless the default symmetric parameter is explicitly set to TRUE, improved formulas are used (Feng & Cliff, 2004) to caculate asymmetric confidence interval (CI) boundary estimates of delta or mean difference: CI(lower/upper)=(d-d^3+-t s_d ((1-2d^2+d^4+t^2 s_d^2)^(-1/2)))/(1-d^2+t^2 s_d^2) with t-values at the given alpha-level taken from Student's t distribution by default (unless the studdist is set FALSE, in which case t-values are based on z-values from the Standard Normal Distribution ). CI_lower/upper=(n-t^2)/(n+t^2), where t is the t-value or z-score at the selected α level (2-tailed) of the respective studdist-controlled distribution, and n the number of observations or cases in the smaller of the two samples. |
[16 or ["CI high", col#] |
Confidence interval upper boundary estimate of delta or mean difference. |
[17 or ["s delta", col#] |
Unbiased sample estimate of the delta standard deviation in column 1. |
[18 or ["var delta", col#] |
Column 1: Variance of delta (unbiased sample estimate), calculated as s_d^2 = (n_y^2*SUM((d_i.-d)^2) + n_x^2*SUM((d_.j-d)^2)) - SUM(SUM((dij-d)^2))) / (n_x*n_y*(n_x-1)*(n_y-1)), or, using the partial variances s_d^2=(n_y^2(n_x-1)s_di.^2 + n_x^2(n_y-1)s_d.j^2 - (n_x n_y -1)s_dij^2)/(n_x n_y (n_x-1) (n_y-1)) , which can also alternatively be put as s_d^2 = (n_y s_di.^2)/(n_x(n_y-1)) + (n_x s_d.j^2)/(n_y(n_x-1)) - (n_x n_y -1)s_dij^2) / (n_x n_y (n_x-1) (n_y-1)). (For differences to Cliff's (1996, p. 138) formula see notes to Row 28 ("var dij") below.) |
[19 or ["se delta", col#] |
Column 2 only: metric Standard error of mean difference: |
[20 or ["z/t score", col#] |
Column 1: z score of delta on the of the respective studdist-controlled distribution (Student's t or standard normal). |
[21 or ["H1 tails p/CI", col#] |
Equals 1 for one-tailed and 2 for two-tailed testing of alternative or H_1-hypothesis, affecting CI and p values. |
[22 or ["p", col#] |
Probability of z/t score (1-sided or 2-sided comparison as shown in row 21). |
[23 or ["Cohen's d", col#] |
Cohen's d effect size estimate of delta. For Cliff's delta inferred from distributional non-overlap as suggested by Grissom & Kim (2005, p. 106 f.) as well as Romano, Kromrey, Coraggio, & Skowronek (2006, p. 14-15), relating to the relative positions of the distributions of X and Y. When Cliff's delta equals 0, there is no effect, and the Y and X distributions overlap completely. If there are effects, a certain percentage of non-overlap between X and Y is created, and the relative positions of the X and Y distribtions shift. The degree of non-overlap thus is a measure of effect size and is expressed as Cohen's d in terms of non-overlap between two normal distributions (based on U1 in Table 2.2.1, Cohen, 1988, p.22). See |
[24 or ["d CI low", col#] |
Column 1: Cohen's d effect size estimate of the lower boundary of confidence interval (row 15) by using the non-overlap strategy. s_d = sqrt(((nx+ny)/(nx ny)) + (d^2/(2(nx+ny)))) . |
[25 or ["d CI high", col#] |
Column 1:Cohen's d estimate of upper boundary of confidence interval (row 16). |
[26 or ["var d.i", col#] |
Row variance of dominance/difference matrix, calculated as |
[27 or ["var dj.", col#] |
Column variance of dominance/difference matrix, calculated as |
[28 or ["var dij", col#] |
Variance of dominance/difference matrix as sample estimate according to Long et al. (2003, section 3.3 before eqn. 67): s_d_ij=SUM(SUM((d_ij-d)^2))/(n_x*n_y-1), thus avoiding Cliff's original (1996, p. 138) suggestion to use (n_x-1)(n_y-1) as the denominator). |
[29 or ["df", col#] |
If the studdist parameter is not set to FALSE, column 1 returns the degrees of freedom (df) used for CI as well as z/t-score and z-probability estimates. |
[30 or ["NNT", col#] |
The number needed to treat effect size (NNT, cf. Cook & Sackett, 1995) is returned based on the delta statistic as delta^{-1} as suggested by Kraemer & Kupfer, 2006, p. 994. |
DEPENDENT/PAIRED GROUPS (paired argument set to TRUE)
In the case of paired data (e.g. pretest-posttest comparisons of the n_x=n_y same subjects), a 4-column-matrix containing 29 rows with values is returned.
The ordinal statistics for d_{ij} can be retrieved from the first three columns (named
within [.,1] |
for the n_x=n_y within-pair changes (where i=j in all cases); |
between [.,2] |
for the overall distribution changes, based on all n^2-n = n(n-1) comparisons where i \ne j, |
combined [.,3] |
for combined inferences d_w + d_b. |
Here, the fourth column (named "metric") contains metric comparison data.
[1 or ["var1_X_pre", col#] |
Original column name of the x (or pretest) input matrix. |
[2 or ["var2_Y_post", col#] |
Original column name of the y (or posttest) input matrix. |
[3 or ["type_title", col#] |
Columns 1-3: Return type of the comparison, in this case "paired". |
[4 or ["N #Y>X", col#] |
Number of occurences (\#) of a posttest observation y_i having a higher value than a pretest observation x_j: N_{\#Y>X}=\#(y_i>x_j) , limited to the respective pairs under observation in within, between or combined. |
[5 or ["N #Y=X", col#] |
Number of occurences of a posttest observation having the same value as a pretest observation, limited to the respective pairs under observation in within, between or combined. |
[6 or ["N #Y<X", col#] |
Number of occurences of a posttest observation having a smaller value than a pretest observation, limited to the respective pairs under observation in within, between or combined. |
[7 or ["PS X>Y", col#] |
Common Language CL effect size or Probability of Superiority (PS) of X over Y (Grissom, 1994,Grissom & Kim, 2005) (limited to the respective pairs under observation in within, between or combined): PS(Y>X)= \#(y_i>x_j)/(n_y n_x) . This effect size reflects the probability that a subject or case randomly chosen from the X- or pre-test-scores under observation has a higher score than than a randomly chosen case from the respective Y- or post-test-subsample (cf. Acion et al., 2006). |
[8 or ["PS Y>X", col#] |
Common Language CL effect size or Probability of Superiority (PS) of Y over X (Grissom, 1994,Grissom & Kim, 2005) (limited to the respective pairs under observation in within, between or combined). |
[9 or ["A X>Y", col#] |
Vargha and Delaney's A as stochastic superiority of X over Y, limited to the respective pairs under observation in within, between or combined. (See codedmes of this orddom package for details.) |
[10 or ["A Y>X", col#] |
Vargha and Delaney's A as stochastic superiority of Y over X, limited to the respective pairs under observation in within, between or combined. (See codedmes of this orddom package for details.) |
[11 or ["delta", col#] |
For columns 1 to 3 ("ordinal"), the respective delta for dependent groups (Cliff, 1996,Long et al., 2003,Feng, 2007) is reported. With d_ij=sign(y_i-x_j), d_w=SUM(SUM(d_ii))/n, where i=j in the n = n_x = n_y possible paired comparisons. d_b=SUM(SUM(d_ij))/(n(n-1)), where i <> j. |
[12 or ["1-alpha", col#] |
Significance or α-level for CI estimation, given as percentage between 0 and 100. |
[13 or ["CI low", col#] |
Confidence interval (CI) lower boundary estimate. Unless the default symmetric parameter is explicitly set to TRUE, asymmetric Confidence interval (CI) boundary estimates for ordinal differences are calculated (Feng & Cliff, 2004; Feng, 2007) as CI(lower/upper)=(d-d^3+-t s_d ((1-2d^2+d^4+t^2 s_d^2)^(-1/2)))/(1-d^2+t^2 s_d^2) , with t-values at the respective significance level based on either Student's t or on z-values from the Standard Normal Distribution, depending on the studdist argument. CI_lower/upper=(n-t^2)/(n+t^2), where t is the t-value or z-score at the selected α level (1- or 2-tailed) of the respective studdist-controlled distribution, and n the number of observations or cases in the smaller of the two samples. |
[14 or ["CI high", col#] |
Confidence interval upper boundary estimate (see row 13). |
[15 or ["s delta", col#] |
Estimated standard deviation of the respective delta statistic. Column 4 reports the metric standard deviation of the paired (within) differences. |
[16 or ["var delta", col#] |
Unbiased estimates of the variances of the respective delta statistic. s_dw^2=SUM((d_ii-d_w)^2)/(n(n-1)). Please note that in various pieces of the available research literature (e.g. Cliff, 1996, eq. 6.8, p. 161), s_{d_w}^{2} is erroneously reported to be calculated as s_dw^2=SUM((d_ii-d_w)^2)/(n-1). The denominator, however must read n(n-1) as "using just (n-1) would give the variance of the individual d_{ii} whereas we want the variance of d_w, which is a kind of mean" (Feng, 07.02.2011, personal communication). |
[17 or ["z/t score", col#] |
z score of delta. In column 4 ("metric") equal to the t-test score (assuming equal variances). |
[18 or ["H1 tails p/CI", col#] |
Equals 1 for one-tailed and 2 for two-tailed testing of alternative or H_1-hypothesis, affecting CI and p values. |
[19 or ["p", col#] |
Probability of z-score (1 or 2-tailed comparison as shown in row 18). |
[20 or ["Cohen's d", col#] |
Cohen's d estimate of the respective delta value (see above). In the metric case, the between group t-value and the original standard deviations are also used for the paired case to avoid overestimation of the effect size (Dunlap et al., 1996). See |
[21 or ["d CI low", col#] |
Column 1 and 2: Cohen's d estimate of lower boundary of the respective confidence interval (row 13) by using the non-overlap calculation strategy. s_d = sqrt(((nx+ny)/(nx ny)) + (d^2/(2(nx+ny)))) . |
[22 or ["d CI high", col#] |
Cohen's d estimate of upper boundary of the respective confidence interval (see row 21 for calculation details). |
[23,3] or ["var d.i",combined] |
Component of s_{d_w+d_b}^2: s_{di.}^2 (Available for the combined analyses in column 3 only.) The metric descriptive in column 4 is the variance of x (or s_x^2. |
[24,3] or ["var dj.",combined] |
Component of s_{d_w+d_b}^2: s_{d.i}^2 (Third column only.) The metric descriptive in column 4 is the variance of y (or s_y^2. |
[25,3] or ["cov(di,dj)",combined] |
Component of s_{d_w+d_b}^2: cov(d_{i.},d_{.j}) (Third column only.) |
[26,3] or ["var dij",combined] |
Component of s_{d_w+d_b}^2: s_{d_{ij}}^2 (Third column only.) |
[27,3] or ["cov(dih,dhi)",combined] |
Component of s_{d_w+d_b}^2: cov(d_{ih},d_{hi}) (Third column only.) |
[28,3] or ["cov(db,dw)",combined] |
Estimated covariance between d_b and d_w: Est[cov(d_b,d_w)] (for purposes of combined inferences). (Third column only.) |
[29 or ["df", col#] |
Unless the studdist argument is not set to FALSE, the degrees of Freedom df used for the CI and z-score calculations are reported in column 1. |
[30 or ["NNT", col#] |
In column 1 and 2, the number needed to treat effect size (NNT, cf. Cook & Sackett, 1995) are returned, based on the underlying delta statistics with NNT= delta^{-1} as suggested by Kraemer & Kupfer, 2006, p. 994. (Column 3 is empty.). |
Jens J. Rogmann, University of Hamburg, Department of Psychology,
Hamburg, Germany (Jens.Rogmann@uni-hamburg.de)
Acion, L., Peterson, J.J., Temple, S., & Arndt, S. (2006). Probabilistic index: an intuitive non-parametric approach to measuring the size of treatment effects. Statistics in Medicine, 25, 591 - 602.
Cliff, N. (1996). Ordinal Methods for Behavioral Data Analysis. Mahwah, NJ: Lawrence Erlbaum.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edition). New York: Academic Press.
Cook, R.J. & Sackett, D.L. (1995). The number needed to treat: A clinically useful measure of treatment effect. British Medical Journal, 310, 452 - 454.
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170 - 177.
Feng, D. (2007). Robustness and Power of Ordinal d for Paired Data. In Shlomo S. Sawilowsky (Ed.), Real Data Analysis (pp. 163-183). Greenwich, CT : Information Age Publishing.
Feng, D., & Cliff, N. (2004). Monte Carlo Evaluation of Ordinal d with Improved Confidence Interval. Journal of Modern Applied Statistical Methods, 3(2), 322-332.
Long, J. D., Feng, D., & Cliff, N. (2003). Ordinal analysis of behavioral data. In J. Schinka & W. F. Velicer (eds.), Research Methods in Psychology. Volume 2 of Handbook of Psychology (I. B. Weiner, Editor-in-Chief). New York: John Wiley & Sons.
Grissom, R.J. (1994). Probability of the superior outcome of one treatment over another. Journal of Applied Psychology, 79, 314-316.
Grissom, R.J. & Kim, J.J. (2005). Effect sizes for research. A broad practical approach. Mahwah, NJ, USA: Erlbaum.
Hedges, L.V. & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA, USA: Academic Press.
Kraemer, H.C. & Kupfer, D.J. (2006). Size of Treatment Effects and Their Importance to Clinical Research and Practice. Biological Psychiatry, 59, 990-996.
McGraw, K.O. & Wong, S.P. (1992). A common language effect size statistic. Psychological Bulletin, 111, 361-365.
Romano, J., Kromrey, J. D., Coraggio, J., & Skowronek, J. (2006). Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen's d for evaluating group differences on the NSSE and other surveys? Paper presented at the annual meeting of the Florida Association of Institutional Research, Feb. 1-3, 2006, Cocoa Beach, Florida. Last retrieved January 2, 2012 from www.florida-air.org/romano06.pdf
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | ## Not run:
#Independent Samples (Data taken from Long et al. (2003), Table 3
## End(Not run)
x<-t(matrix(c(3,3,3,4,5,6,12,12,13,14,15,15,15,15,15,16,18,18,18,23,23,27,28,28,43),1))
colnames(x)<-c("Nonalcohol.")
y<-t(matrix(c(1,4,6,7,7,14,14,18,19,20,21,24,25,26,26,26,27,28,28,30,33,33,44,45,50),1))
colnames(y)<-c("Alcoholic")
orddom(x,y,paired=FALSE,outputfile="tmp_r.txt")
## Not run:
#Paired Comparison with data written to file (Data taken from Long et al. (2003), Table 4
## End(Not run)
x<-t(matrix(c(2,6,6,7,7,8,8,9,9,9,10,10,10,11,11,12,13,14,15,16),1))
colnames(x)<-c("Incidental")
y<-t(matrix(c(4,11,8,9,10,11,11,5,14,12,13,10,14,16,14,13,15,15,16,10),1))
colnames(y)<-c("Intentional")
orddom_f(y,x,paired=TRUE,symmetric=FALSE)
## Not run:
#Directly returns d_b of the paired comparison
## End(Not run)
orddom(x,y,,TRUE,,,)[11,2]
|
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.