evaluate: Model Evaluation

In trinker/termco: Counts of Terms and Substrings

Description

Get accuracy, precision, and recall for multi-class, multi-tag, predictions.

Usage

evaluate(x, known)

Arguments

x	The model classification list/vector (typically the results of classify).
known	The known expert coded list/vector of outcomes.

Value

Returns a list of seven elements:

N	The number of elements being assessed
confusion_matrix	A list of confusion matrices for each tag
tag_accuracy	A tag named vector of accuracies computed from the confusion matrices; (tp + tn)/(tp + tn + fp + fn)
tag_precision	A tag named vector of precisions computed from the confusion matrices; tp/(tp + fp)
tag_recall	A tag named vector of accuracies computed from the confusion matrices; tp/(tp + fn)
macro_averaged	Macro averaged accuracy, precision, and recall; computed accuracy, precision, and recall for each confusion matrix and average
micro_averaged	Micro averaged accuracy, precision, and recall; add the confusion amtrices and compute accuracy, precision, and recall

References

https://www.youtube.com/watch?v=OwwdYHWRB5E&index=31&list=PL6397E4B26D00A269

Examples

known <- list(1:3, 3, NA, 4:5, 2:4, 5, integer(0))
tagged <- list(1:3, 3, 4, 5:4, c(2, 4:3), 5, integer(0))
evaluate(tagged, known)

## Examples
library(dplyr)
data(presidential_debates_2012)

discoure_markers <- list(
    response_cries = c("\\boh", "\\bah", "\\baha", "\\bouch", "yuk"),
    back_channels = c("uh[- ]huh", "uhuh", "yeah"),
    summons = "hey",
    justification = "because"
)


## Only Single Tag Allowed Per Text Element
mod1 <- presidential_debates_2012 %>%
    with(., term_count(dialogue, TRUE, discoure_markers)) %>%
    classify()

fake_known <- mod1
set.seed(1)
fake_known[sample(1:length(fake_known), 300)] <- "random noise"

evaluate(mod1, fake_known)

## Multiple Tags Allowed
mod2 <- presidential_debates_2012 %>%
    with(., term_count(dialogue, TRUE, discoure_markers)) %>%
    classify(n = 2)

fake_known2 <- mod2
set.seed(30)
fake_known2[sample(1:length(fake_known2), 500)] <- c("random noise", "back_channels")

(myacc <- evaluate(mod2, fake_known2))
myacc$confusion_matrix
myacc$tag_accuracy

trinker/termco documentation built on Jan. 7, 2022, 3:32 a.m.