classify-package: Classification Accuracy and Consistency under IRT models.
In classify: Classification Accuracy and Consistency under IRT models.
Description
Details
Author(s)
References
Examples
Description
IRT classification uses the probability that candidates of a given ability, will answer correctly questions of a specified difficulty to calculate the probability of their achieving every possible score in a test. Due to the IRT assumption of conditional independence (that is every answer given is assumed to depend only on the latent trait being measured) the probability of candidates achieving these potential scores can be expressed by multiplication of probabilities for item responses for a given ability. Once the true score and the probabilities of achieving all other scores have been determined for a candidate the probability of their score lying in the same category as that of their true score (classification accuracy), or the probability of consistent classification in a category over administrations (classification consistency), can be calculated.
Details
Package: classify
Type: Package
Version: 1.0
Date: 2012-04-30
License: GPL (>= 2)
Depends: Rcpp (>= 0.9.10), plyr, ggplot2, R2WinBUGS, lattice, reshape2,
methods, R2jags
LinkingTo: Rcpp
Packaged: 2012-06-08 12:43:25 UTC; cwheadon
Built: R 2.15.0; i386-pc-mingw32; 2012-06-08 12:43:33 UTC; windows
Index:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
beta.list Extracts Beta Values from Bugs Sims File
biology Polytomous Responses from 200 Candidates to 31
Questions
classification-class Class '"classification"'
classify Calculate Classification Statistics
classify-package Classification Accuracy and Consistency under
IRT Models
classify.bug Classification Accuracy and Consistency from
Bugs Replicate Parameters
gpcm Generalised Partial Credit Model Derived
Probabilities
gpcm.bug Extracts IRT Model Parameters from Bugs Models
gpcm.rc IRT Derived Predicted Conditional Number
Correct Score Distribution
pcm Partial Credit Model Derived Probabilities
physics Dichotomous Responses from 200 Candidates to 25
Questions
plot.scores Plot Methods for Classification and Scores
Objects
rasch Rasch Derived Probabilities
scores-class Class '"scores"'
scores.gpcm.bug Expected and Conditional Summed Score
Distributions
summary-methods Summary Stats for Classification Accuracy and
Consistency
thpl Three Parameter IRT Model Derived Probabilities
tpl Two Parameter IRT model Derived Probabilities
wlord Lord and Wingersky Recursion Formula
Author(s)
Dr Chris Wheadon and Dr Ian Stockford
Maintainer: Dr Chris Wheadon <chris.wheadon@gmail.com>
References
Curtis, S.M.(2010) BUGS Code for Item Response Theory, Journal of Statistical Software, Code Snippets, 36(1),1–34.
Hanson, B.A., Beguin, A.A.(2002) Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26, 3–24.
Kolen M.J., Brennan, R.L. (2004) Test Equating, Scaling, and Linking. Statistics in Social Science and Public Policy.
Lee, W. (2008) Classification consistency and accuracy for complex assessments using item response theory, (No. 27) CASMA Research Report Iowa City, IA: Center for Advanced Studies in Measurement and Assessment, University of Iowa.
Lord, F., Wingersky, M. (1984) Comparison of IRT true-score and equipercentile observed-score equatings, Applied Psychological Measurement, 8, 452–461.
Lunn, D.J., Thomas, A., Best, N., Spiegelhalter, D. (2000) WinBUGS, A Bayesian modelling framework: concepts, structure, and extensibility. Statistics and Computing,
10, 325–337.
Plummer, M (2012) Just Another Gibbs Sampler, version 3.2.0 http://mcmcjags.sourceforge.net/
Swaminathan, H., Hambleton, R. K., Rogers, H. J. (2007) Assessing the fit of Item Response Theory models. In C. R. Rao, S. Sinharay (Eds.), Handbook of statistics, Vol. 26, 683–718.
Wheadon, C., (2014) Classification Accuracy and Consistency under Item Response Theory Models Using the Package classify, Journal of Statistical Software, 56(10) 1–14.
Wheadon, C., Stockford, I., (2011) Classification accuracy and consistency in GCSE and A level examinations offered by the Assessment and Qualifications Alliance (AQA) November 2008 to June 2009, Ofqual's Reliability Compendium. Office of Qualification and Examinations Regulation.
Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
## Not run:
## Rasch or 2pl Model
# Data should be a numeric matrix, with one row per candidate, one column per item
data(physics)
# If reading from csv, the following is recommended:
# physics<-read.csv("physics.csv", header=TRUE, sep=",", na.strings = "-")
# physics<-physics[complete.cases(physics),]
# physics<-physics[sample(1:nrow(physics), 200, replace=FALSE),]
# physics<-as.matrix(physics)
n <- nrow(physics)
p <- ncol(physics)
# Boundary marks in ascending order
bnds <- c(9,11,13,15,18,21)
# Labels for boundaries (one more boundary than label)
lbls <- c("U","E","D","C","B","A","A*")
# Specify bugs file - included in the classify/bugs directory
# 2 pl model
mdl <- "tpl.bug"
# Rasch model
# mdl <- "rasch.bug"
# Item marks
Y <- physics
# Mean and standard deviation of delta
m.delta <- 0.0
s.delta <- 1.0
# Mean and standard deviation of alpha, comment out for Rasch model
m.alpha <- 0.0
s.alpha <- 1.0
# Data set for the 2 pl model
data <- list("Y", "n", "p", "m.delta", "s.delta", "m.alpha", "s.alpha")
# Parameters to monitor for the 2 pl model
monitor <- c("delta", "theta", "alpha")
# Rasch model
# data <- list("Y", "n", "p", "m.delta", "s.delta")
# monitor <- c("delta", "theta")
# Set to location of bug file
jags.file <- file.path(getwd(), "R/R-2.15.0/library/classify/bugs" ,mdl)
# JAGS
# may require set.seed(1234) depending on version of R
system.time(jagsout <- jags(data=data, inits=NULL, parameters.to.save=monitor,
model.file=jags.file,
n.iter=2000, n.thin=10, n.burnin=1000))
sims <- jagsout$BUGSoutput$sims.matrix
# Bugs
# Change this to your bugs directory
# bugs.directory = "C:/Program Files/WinBUGS14"
# system.time(bugsout <- bugs(data=data, inits=NULL, parameters.to.save=monitor,
# model.file=jags.file,
# n.iter=2000, n.thin=10, n.burnin=1000))
# sims <- bugsout$sims.matrix
# Estimate conditional score and expected score distributions
scores <- scores.gpcm.bug(Y,sims,mdl)
# Plots
plot(scores)
# Save plot
# ggsave("expected.pdf")
plot(scores,type="cond")
plot(scores,type="qq",alpha=0.5)
# Estimate accuracy statistics
accs <- classify.bug(sims,scores,bnds,lbls)
summary(accs)
plot(accs)
plot(accs,type="kappa")
plot(accs,type="density")
##############################################
##############################################
## PCM or GPCM models
# Data should be a numeric matrix, with one row per candidate, one column per item
data(biology)
# If reading from csv, the following is recommended:
# biology<-read.csv("biology.csv", header=TRUE, sep=",", na.strings = "-")
# biology<-biology[complete.cases(biology),]
# biology<-biology[sample(1:nrow(biology), 200, replace=FALSE),]
# biology<-as.matrix(biology)
n <- nrow(biology)
p <- ncol(biology)
# Boundary marks in ascending order
bnds <- c(26,30,35,40,45)
# Labels for boundaries (one more boundary than label)
lbls <- c("U","E","D","C","B","A")
# Specify bugs file - included in the classify/bugs directory
# GPCM
mdl <- "gpcm.bug"
# PCM
#mdl <- "pcm.bug"
# Bugs polytomous models require polytomous scores as categories
Y <- biology + 1
# Specify response categories
K <- as.numeric(apply(Y,2,max,na.rm = TRUE))
# Mean and standard deviation of alpha and beta parameters
m.beta <- 0.0
s.beta <- 1.0
# Comment out for PCM
m.alpha <- 0.0
s.alpha <- 1.0
# GPCM
data <- list("Y", "n", "p", "K",
"m.beta", "s.beta", "m.alpha", "s.alpha")
monitor <- c("beta", "theta", "alpha")
# PCM
#data <- list("Y", "n", "p", "K",
# "m.beta", "s.beta")
#monitor <- c("beta", "theta")
# Initial values for beta set to 0, matrix padded with NA
beta <- t(sapply(1:p, function(j) c(rep(NA, K[j]), rep(0.0, max(K) - K[j]))))
data <- c(data, "beta")
# Change this to bugs file directory
jags.file <- file.path(getwd(), "R/R-2.15.0/library/classify/bugs" ,mdl)
# Simulations and sampling
iter <- 2000
burnin <- 1000
thin <- 10
## JAGS
estimation <- "jags"
# may require set.seed(1234) depending on version of R
system.time(jagsout <- jags(data=data, inits=NULL, parameters.to.save=monitor,
model.file=jags.file,
n.iter=iter, n.thin=thin, n.burnin=burnin))
sims <- jagsout$BUGSoutput$sims.matrix
## Bugs
#estimation <- "bugs"
# Change this to your bugs directory
#bugs.directory = "C:/Program Files/WinBUGS14"
#system.time(bugsout <- jags(data=data, inits=NULL, parameters.to.save=monitor,
# model.file=jags.file,
# n.iter=iter, n.thin=thin, n.burnin=burnin))
#sims <- bugsout$sims.matrix
# Estimate conditional score and expected score distributions
scores <- scores.gpcm.bug(biology,sims,mdl)
# Plots
plot(scores)
# Save plot
#ggsave("expected.pdf")
plot(scores,type="cond")
plot(scores,type="qq",alpha=0.5)
# Estimate accuracy statistics
accs <- classify.bug(sims,scores,bnds,lbls)
summary(accs)
plot(accs)
plot(accs,type="kappa")
plot(accs,type="density")
## End(Not run)
classify documentation built on May 29, 2017, 7:49 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Description Details Author(s) References Examples
Description
IRT classification uses the probability that candidates of a given ability, will answer correctly questions of a specified difficulty to calculate the probability of their achieving every possible score in a test. Due to the IRT assumption of conditional independence (that is every answer given is assumed to depend only on the latent trait being measured) the probability of candidates achieving these potential scores can be expressed by multiplication of probabilities for item responses for a given ability. Once the true score and the probabilities of achieving all other scores have been determined for a candidate the probability of their score lying in the same category as that of their true score (classification accuracy), or the probability of consistent classification in a category over administrations (classification consistency), can be calculated.
Details
| Package: | classify |
| Type: | Package |
| Version: | 1.0 |
| Date: | 2012-04-30 |
| License: | GPL (>= 2) |
| Depends: | Rcpp (>= 0.9.10), plyr, ggplot2, R2WinBUGS, lattice, reshape2, methods, R2jags |
| LinkingTo: | Rcpp |
| Packaged: | 2012-06-08 12:43:25 UTC; cwheadon |
| Built: | R 2.15.0; i386-pc-mingw32; 2012-06-08 12:43:33 UTC; windows |
Index:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | beta.list Extracts Beta Values from Bugs Sims File
biology Polytomous Responses from 200 Candidates to 31
Questions
classification-class Class '"classification"'
classify Calculate Classification Statistics
classify-package Classification Accuracy and Consistency under
IRT Models
classify.bug Classification Accuracy and Consistency from
Bugs Replicate Parameters
gpcm Generalised Partial Credit Model Derived
Probabilities
gpcm.bug Extracts IRT Model Parameters from Bugs Models
gpcm.rc IRT Derived Predicted Conditional Number
Correct Score Distribution
pcm Partial Credit Model Derived Probabilities
physics Dichotomous Responses from 200 Candidates to 25
Questions
plot.scores Plot Methods for Classification and Scores
Objects
rasch Rasch Derived Probabilities
scores-class Class '"scores"'
scores.gpcm.bug Expected and Conditional Summed Score
Distributions
summary-methods Summary Stats for Classification Accuracy and
Consistency
thpl Three Parameter IRT Model Derived Probabilities
tpl Two Parameter IRT model Derived Probabilities
wlord Lord and Wingersky Recursion Formula
|
Author(s)
Dr Chris Wheadon and Dr Ian Stockford
Maintainer: Dr Chris Wheadon <chris.wheadon@gmail.com>
References
Curtis, S.M.(2010) BUGS Code for Item Response Theory, Journal of Statistical Software, Code Snippets, 36(1),1–34.
Hanson, B.A., Beguin, A.A.(2002) Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26, 3–24.
Kolen M.J., Brennan, R.L. (2004) Test Equating, Scaling, and Linking. Statistics in Social Science and Public Policy.
Lee, W. (2008) Classification consistency and accuracy for complex assessments using item response theory, (No. 27) CASMA Research Report Iowa City, IA: Center for Advanced Studies in Measurement and Assessment, University of Iowa.
Lord, F., Wingersky, M. (1984) Comparison of IRT true-score and equipercentile observed-score equatings, Applied Psychological Measurement, 8, 452–461.
Lunn, D.J., Thomas, A., Best, N., Spiegelhalter, D. (2000) WinBUGS, A Bayesian modelling framework: concepts, structure, and extensibility. Statistics and Computing, 10, 325–337.
Plummer, M (2012) Just Another Gibbs Sampler, version 3.2.0 http://mcmcjags.sourceforge.net/
Swaminathan, H., Hambleton, R. K., Rogers, H. J. (2007) Assessing the fit of Item Response Theory models. In C. R. Rao, S. Sinharay (Eds.), Handbook of statistics, Vol. 26, 683–718.
Wheadon, C., (2014) Classification Accuracy and Consistency under Item Response Theory Models Using the Package classify, Journal of Statistical Software, 56(10) 1–14.
Wheadon, C., Stockford, I., (2011) Classification accuracy and consistency in GCSE and A level examinations offered by the Assessment and Qualifications Alliance (AQA) November 2008 to June 2009, Ofqual's Reliability Compendium. Office of Qualification and Examinations Regulation.
Examples
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 | ## Not run:
## Rasch or 2pl Model
# Data should be a numeric matrix, with one row per candidate, one column per item
data(physics)
# If reading from csv, the following is recommended:
# physics<-read.csv("physics.csv", header=TRUE, sep=",", na.strings = "-")
# physics<-physics[complete.cases(physics),]
# physics<-physics[sample(1:nrow(physics), 200, replace=FALSE),]
# physics<-as.matrix(physics)
n <- nrow(physics)
p <- ncol(physics)
# Boundary marks in ascending order
bnds <- c(9,11,13,15,18,21)
# Labels for boundaries (one more boundary than label)
lbls <- c("U","E","D","C","B","A","A*")
# Specify bugs file - included in the classify/bugs directory
# 2 pl model
mdl <- "tpl.bug"
# Rasch model
# mdl <- "rasch.bug"
# Item marks
Y <- physics
# Mean and standard deviation of delta
m.delta <- 0.0
s.delta <- 1.0
# Mean and standard deviation of alpha, comment out for Rasch model
m.alpha <- 0.0
s.alpha <- 1.0
# Data set for the 2 pl model
data <- list("Y", "n", "p", "m.delta", "s.delta", "m.alpha", "s.alpha")
# Parameters to monitor for the 2 pl model
monitor <- c("delta", "theta", "alpha")
# Rasch model
# data <- list("Y", "n", "p", "m.delta", "s.delta")
# monitor <- c("delta", "theta")
# Set to location of bug file
jags.file <- file.path(getwd(), "R/R-2.15.0/library/classify/bugs" ,mdl)
# JAGS
# may require set.seed(1234) depending on version of R
system.time(jagsout <- jags(data=data, inits=NULL, parameters.to.save=monitor,
model.file=jags.file,
n.iter=2000, n.thin=10, n.burnin=1000))
sims <- jagsout$BUGSoutput$sims.matrix
# Bugs
# Change this to your bugs directory
# bugs.directory = "C:/Program Files/WinBUGS14"
# system.time(bugsout <- bugs(data=data, inits=NULL, parameters.to.save=monitor,
# model.file=jags.file,
# n.iter=2000, n.thin=10, n.burnin=1000))
# sims <- bugsout$sims.matrix
# Estimate conditional score and expected score distributions
scores <- scores.gpcm.bug(Y,sims,mdl)
# Plots
plot(scores)
# Save plot
# ggsave("expected.pdf")
plot(scores,type="cond")
plot(scores,type="qq",alpha=0.5)
# Estimate accuracy statistics
accs <- classify.bug(sims,scores,bnds,lbls)
summary(accs)
plot(accs)
plot(accs,type="kappa")
plot(accs,type="density")
##############################################
##############################################
## PCM or GPCM models
# Data should be a numeric matrix, with one row per candidate, one column per item
data(biology)
# If reading from csv, the following is recommended:
# biology<-read.csv("biology.csv", header=TRUE, sep=",", na.strings = "-")
# biology<-biology[complete.cases(biology),]
# biology<-biology[sample(1:nrow(biology), 200, replace=FALSE),]
# biology<-as.matrix(biology)
n <- nrow(biology)
p <- ncol(biology)
# Boundary marks in ascending order
bnds <- c(26,30,35,40,45)
# Labels for boundaries (one more boundary than label)
lbls <- c("U","E","D","C","B","A")
# Specify bugs file - included in the classify/bugs directory
# GPCM
mdl <- "gpcm.bug"
# PCM
#mdl <- "pcm.bug"
# Bugs polytomous models require polytomous scores as categories
Y <- biology + 1
# Specify response categories
K <- as.numeric(apply(Y,2,max,na.rm = TRUE))
# Mean and standard deviation of alpha and beta parameters
m.beta <- 0.0
s.beta <- 1.0
# Comment out for PCM
m.alpha <- 0.0
s.alpha <- 1.0
# GPCM
data <- list("Y", "n", "p", "K",
"m.beta", "s.beta", "m.alpha", "s.alpha")
monitor <- c("beta", "theta", "alpha")
# PCM
#data <- list("Y", "n", "p", "K",
# "m.beta", "s.beta")
#monitor <- c("beta", "theta")
# Initial values for beta set to 0, matrix padded with NA
beta <- t(sapply(1:p, function(j) c(rep(NA, K[j]), rep(0.0, max(K) - K[j]))))
data <- c(data, "beta")
# Change this to bugs file directory
jags.file <- file.path(getwd(), "R/R-2.15.0/library/classify/bugs" ,mdl)
# Simulations and sampling
iter <- 2000
burnin <- 1000
thin <- 10
## JAGS
estimation <- "jags"
# may require set.seed(1234) depending on version of R
system.time(jagsout <- jags(data=data, inits=NULL, parameters.to.save=monitor,
model.file=jags.file,
n.iter=iter, n.thin=thin, n.burnin=burnin))
sims <- jagsout$BUGSoutput$sims.matrix
## Bugs
#estimation <- "bugs"
# Change this to your bugs directory
#bugs.directory = "C:/Program Files/WinBUGS14"
#system.time(bugsout <- jags(data=data, inits=NULL, parameters.to.save=monitor,
# model.file=jags.file,
# n.iter=iter, n.thin=thin, n.burnin=burnin))
#sims <- bugsout$sims.matrix
# Estimate conditional score and expected score distributions
scores <- scores.gpcm.bug(biology,sims,mdl)
# Plots
plot(scores)
# Save plot
#ggsave("expected.pdf")
plot(scores,type="cond")
plot(scores,type="qq",alpha=0.5)
# Estimate accuracy statistics
accs <- classify.bug(sims,scores,bnds,lbls)
summary(accs)
plot(accs)
plot(accs,type="kappa")
plot(accs,type="density")
## End(Not run)
|
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.