get_coder_metrics: Calculate reliability measures based on content analysis

In aifeducation: Artificial Intelligence for Education

Calculate reliability measures based on content analysis

Description

This function calculates different reliability measures which are based on the empirical research method of content analysis.

Usage

get_coder_metrics(
  true_values = NULL,
  predicted_values = NULL,
  return_names_only = FALSE
)

Arguments

true_values	factor containing the true labels/categories.
predicted_values	factor containing the predicted labels/categories.
return_names_only	bool If TRUE returns only the names of the resulting vector. Use FALSE to request computation of the values.

Value

If return_names_only = FALSE returns a vector with the following reliability measures:

• iota_index: Iota Index from the Iota Reliability Concept Version 2.

• min_iota2: Minimal Iota from Iota Reliability Concept Version 2.

• avg_iota2: Average Iota from Iota Reliability Concept Version 2.

• max_iota2: Maximum Iota from Iota Reliability Concept Version 2.

• min_alpha: Minmal Alpha Reliability from Iota Reliability Concept Version 2.

• avg_alpha: Average Alpha Reliability from Iota Reliability Concept Version 2.

• max_alpha: Maximum Alpha Reliability from Iota Reliability Concept Version 2.

• static_iota_index: Static Iota Index from Iota Reliability Concept Version 2.

• dynamic_iota_index: Dynamic Iota Index Iota Reliability Concept Version 2.

• kalpha_nominal: Krippendorff's Alpha for nominal variables.

• kalpha_ordinal: Krippendorff's Alpha for ordinal variables.

• kendall: Kendall's coefficient of concordance W with correction for ties.

• c_kappa_unweighted: Cohen's Kappa unweighted.

• c_kappa_linear: Weighted Cohen's Kappa with linear increasing weights.

• c_kappa_squared: Weighted Cohen's Kappa with quadratic increasing weights.

• kappa_fleiss: Fleiss' Kappa for multiple raters without exact estimation.

• percentage_agreement: Percentage Agreement.

• balanced_accuracy: Average accuracy within each class.

• gwet_ac: Gwet's AC1/AC2 agreement coefficient.

If return_names_only = TRUE returns only the names of the vector elements.

See Also

Other classifier_utils: calc_standard_classification_measures()

aifeducation documentation built on April 4, 2025, 2:01 a.m.