evaluate: Model Evaluation

Description Usage Arguments Value References Examples

View source: R/evaluate.R

Description

Get accuracy, precision, and recall for multi-class, multi-tag, predictions.

Usage

1
evaluate(x, known)

Arguments

x

The model classification list/vector (typically the results of classify).

known

The known expert coded list/vector of outcomes.

Value

Returns a list of seven elements:

N

The number of elements being assessed

confusion_matrix

A list of confusion matrices for each tag

tag_accuracy

A tag named vector of accuracies computed from the confusion matrices; (tp + tn)/(tp + tn + fp + fn)

tag_precision

A tag named vector of precisions computed from the confusion matrices; tp/(tp + fp)

tag_recall

A tag named vector of accuracies computed from the confusion matrices; tp/(tp + fn)

macro_averaged

Macro averaged accuracy, precision, and recall; computed accuracy, precision, and recall for each confusion matrix and average

micro_averaged

Micro averaged accuracy, precision, and recall; add the confusion amtrices and compute accuracy, precision, and recall

References

https://www.youtube.com/watch?v=OwwdYHWRB5E&index=31&list=PL6397E4B26D00A269

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
known <- list(1:3, 3, NA, 4:5, 2:4, 5, integer(0))
tagged <- list(1:3, 3, 4, 5:4, c(2, 4:3), 5, integer(0))
evaluate(tagged, known)

## Examples
library(dplyr)
data(presidential_debates_2012)

discoure_markers <- list(
    response_cries = c("\\boh", "\\bah", "\\baha", "\\bouch", "yuk"),
    back_channels = c("uh[- ]huh", "uhuh", "yeah"),
    summons = "hey",
    justification = "because"
)


## Only Single Tag Allowed Per Text Element
mod1 <- presidential_debates_2012 %>%
    with(., term_count(dialogue, TRUE, discoure_markers)) %>%
    classify()

fake_known <- mod1
set.seed(1)
fake_known[sample(1:length(fake_known), 300)] <- "random noise"

evaluate(mod1, fake_known)

## Multiple Tags Allowed
mod2 <- presidential_debates_2012 %>%
    with(., term_count(dialogue, TRUE, discoure_markers)) %>%
    classify(n = 2)

fake_known2 <- mod2
set.seed(30)
fake_known2[sample(1:length(fake_known2), 500)] <- c("random noise", "back_channels")

(myacc <- evaluate(mod2, fake_known2))
myacc$confusion_matrix
myacc$tag_accuracy

trinker/termco documentation built on Jan. 7, 2022, 3:32 a.m.