validate_model: Manual Assessment of a Model
In trinker/termco: Counts of Terms and Substrings

Description Usage Arguments Value Note Examples

validate_model - Check how well a regex model is tagging using human interaction to assess the model.

assign_validation_task - Create human assignments to assess how well a model is functioning. The coder can use the correct column to assess how well the tag fits the text columns.

validate_model(x, n = 20, width = 50, tags = 1, filter = NULL, ...)

assign_validation_task(
  x,
  n = 20,
  checks = 1,
  coders = "coder",
  out = NULL,
  as.list = TRUE,
  ...
)

`x`	A `term_count` model object (i.e., `grouping.var = TRUE` was used in `term_count`).
`n`	The number of samples to take from each regex tag assignment. Tags with less than `n` will use the full number available.
`width`	The width of the text display.
`tags`	The number of classifications per row/element to allow. Ties are broken probabilistically by default.
`filter`	Validate a subset of the original tags. Useful for when the user finds a mistake in the classifier and wants to retest only portions of the model.
`checks`	The number of coders needed per tag assignment.
`coders`	A vector of coders to assign tasks to.
`out`	A directory name to create and output csv file(s) to.
`as.list`	logical. Should the assignments be dsplayed as a list of `data.frame` or as a single `data.frame`?
`...`	Other arguments passed to `classify`.

validate_model - Returns a data.frame of the class 'validate_model'. Note that the pretty print is a tag summarized version of the model accuracy standard error, and confidence intervals from summary.validate_model. n.tagged is the number of potential tags from the column sums of the termco object whereas n.classified is the number of elements actually classified into that tag group by the classify function.

assign_validation_task - Returns a data.frame/.csv or list of data.frames/.csvs. Columns in the data.frames include:

`coder`	The assgned coder (person for the task).
`index`	The row/element number of the text.
`correct`	A blank column for coders to dummy/logical code if the tag assignment for that text was accurate.
`tag`	The tag that was assigned to the text.
`text`	The text to which the tag was assigned.

This function assigns tags using the classify function. One element may recieve multiple tags.

## Not run: 
data(presidential_debates_2012)

discoure_markers <- list(
    response_cries = c("\\boh", "\\bah", "\\baha", "\\bouch", "yuk"),
    back_channels = c("uh[- ]huh", "uhuh", "yeah"),
    summons = "hey",
    justification = "because"
)

## A model (note: `grouping.var = TRUE` to make a model)
(x <- with(presidential_debates_2012,
    term_count(dialogue, grouping.var = TRUE, term.list = discoure_markers)
))

## Requires interaction
out <- validate_model(x)
out
plot(out)

## Validate a subset of the model
out2 <- validate_model(x, filter = c('response_cries', 'summons'))
out2
plot(out2)

## Assign tasks externally
assign_validation_task(x, checks = 3,
    coders = c('fred', 'jade', 'sally', 'jim', 'shelly'), out='testing')
assign_validation_task(x, checks = 3,
    coders = c('fred', 'jade', 'sally', 'jim', 'shelly'), as.list = FALSE,
    out='testing2')

## End(Not run)