estimateMetaI: Estimate Measures of Metacognition from Information Theory

In statConfR: Models of Decision Confidence and Measures of Metacognition

Estimate Measures of Metacognition from Information Theory

Description

estimateMetaI estimates meta-I, an information-theoretic measure of metacognitive sensitivity proposed by Dayan (2023), as well as similar derived measures, including meta-I_{1}^{r} and Meta-I_{2}^{r}. These are different normalizations of meta-I:

• Meta-I_{1}^{r} normalizes by the meta-I that would be expected from an underlying normal distribution with the same sensitivity.

• Meta-I_{1}^{r\prime} is a variant of meta-I_{1}^{r} not discussed by Dayan (2023) which normalizes by the meta-I that would be expected from an underlying normal distribution with the same accuracy (this is similar to the sensitivity approach but without considering variable thresholds).

• Meta-I_{2}^{r} normalizes by the maximum amount of meta-I which would be reached if all uncertainty about the stimulus was removed.

• RMI normalizes meta-I by the range of its possible values and therefore scales between 0 and 1. RMI is a novel measure not discussed by Dayan (2023).

All measures can be calculated with a bias-reduced variant for which the observed frequencies are taken as underlying probability distribution to estimate the sampling bias. The estimated bias is then subtracted from the initial measures. This approach uses Monte-Carlo simulations and is therefore not deterministic (values can vary from one evaluation of the function to the next). However, this is a simple way to reduce the bias inherent in these measures.

Usage

estimateMetaI(data, bias_reduction = TRUE)

Arguments

data

a data.frame where each row is one trial, containing following variables: participant (some group ID, most often a participant identifier; the meta-I measures are estimated for each subset of data determined by the different values of this column), stimulus (stimulus category in a binary choice task, should be a factor with two levels, otherwise it will be transformed to a factor with a warning), rating (discrete confidence judgments, should be a factor with levels ordered from lowest confidence to highest confidence; otherwise will be transformed to factor with a warning), correct (encoding whether the response was correct; should be 0 for incorrect responses and 1 for correct responses)

bias_reduction

logical. Whether to apply the bias reduction or not. If runtime is too long, consider setting this to FALSE (default: TRUE).

Details

It is assumed that a classifier (possibly a human being performing a discrimination task) or an algorithmic classifier in a classification application, makes a binary prediction R about a true state of the world S and gives a confidence rating C. Meta-I is defined as the mutual information between the confidence and accuracy and is calculated as the transmitted information minus the minimal information given the accuracy,

meta-I = I(S; R, C) - I(S; R).

This is equivalent to Dayan's formulation where meta-I is the information that confidence transmits about the correctness of a response,

meta-I = I(S = R; C).

Meta-I is expressed in bits, i.e. the log base is 2). The other measures are different normalizations of meta-I and are unitless. It should be noted that Dayan (2023) pointed out that a liberal or conservative use of the confidence levels will affected the mutual information and thus influence meta-I.

Value

a data.frame with one row for each subject and the following columns:

• participant is the participant ID,

• meta_I is the estimated meta-I value (expressed in bits, i.e. log base is 2),

• meta_Ir1 is meta-I_{1}^{r},

• meta_Ir1_acc is meta-I_{1}^{r\prime},

• meta_Ir2 is meta-I_{2}^{r}, and

• RMI is RMI.

Author(s)

Sascha Meyen, [e-mail]

References

Dayan, P. (2023). Metacognitive Information Theory. Open Mind, 7, 392–411. doi:10.1162/opmi_a_00091

Examples

# 1. Select two subjects from the masked orientation discrimination experiment
data <- subset(MaskOri, participant %in% c(1:2))
head(data)

# 2. Calculate meta-I measures with bias reduction (this may take 10 s per subject)

metaIMeasures <- estimateMetaI(data)


# 3. Calculate meta-I measures for all participants without bias reduction (much faster)
metaIMeasures <- estimateMetaI(MaskOri, bias_reduction = FALSE)
metaIMeasures

statConfR documentation built on April 3, 2025, 5:35 p.m.