wlin.conc: Inter-rater agreement among a set of raters for ordinal data...

Description Usage Arguments Value Author(s) References Examples


Computes a statistic as an index of inter-rater agreement among a set of raters in case of ordinal data using linear weights. The matrix of linear weights is defined inside the function. This procedure is based on a statistic not affected by Kappa paradoxes. It is also possible to get the confidence interval at level alpha using the percentile Bootstrap and to evaluate if the agreement is nil using the Monte Carlo algorithm. Fleiss' Kappa cannot be used in case of ordinal data. It is advisable to use set.seed to get the same replications for Bootstrap confidence limits and Montecarlo test.


wlin.conc(db, test = "Default", B = 1000, alpha = 0.05)



n*c matrix or data frame, n subjects c categories. The numbers inside the matrix or data frame indicate how many raters chose a specific category for a given subject. A sum of row indicates the total number of raters who evaluated a given subject. In case of ordinal data, the c categories can be sorted according to a specific scale.


Statistical test to evaluate if the raters make random assignment regardless of the characteristic of each subject. Under null hypothesis, it corresponds to a high percentage of assignment errors. Thus, the expected agreement is weak. If this argument is not specified the p value are not being computed.


Number of iterations for the percentile Bootstrap and for Monte Carlo test.


Level of significance for Bootstrap confidence interval and for Monte Carlo algorithm if it is specified


A list containing the following components:


A list with the index of inter-rater agreement not affected by Kappa paradoxes for ordinal data and the percentile Bootstrap confidence interval. If the test argument is specified the p value is also shown.


Piero Quatto piero.quatto@unimib.it, Daniele Giardiello daniele.giardiello1@gmail.com Stefano Vigliani stefano.vigliani@izsler.it


Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 378-382

Falotico, R. Quatto, P. (2010). On avoiding paradoxes in assessing inter-rater agreement. Italian Journal of Applied Statistics 22, 151-160

Marasini, D. Quatto, P. Ripamonti, E. (2014). Assessing the inter-rater agreement for ordinal data through weighted indexes. Statistical methods in medical research.



Search within the raters package
Search all R packages, documentation and source code

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.