In mufcItay/weaknull: A Nonparametric Frequentist Statistical Framework to Test Nondirectional Null Hypotheses

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)

signcon

The goal of signcon is to provide a frequentist statistical framework to test weak, nondirectional null hypotheses. Weak null hypotheses may be rejected in cases where individual subjects show reliable effects, even if these effects have opposite signs and cancel out at the group level in a standard t-test.

Installation

You can install the development version of signcon like so:

remotes::install_github('mufcItay/signcon')

Example

To demonstrate the use of signcon, we use confidence rating data from an experiment where participants reported the orientation of visual gratings, and then rated their confidence on a 1-6 scale. We ask whether confidence levels were similar for the two responses, first using a standard t-test, and then using non-directional sign-consistency and classification tests:

library(signcon)
library(magrittr)

# compare the mean confidence for the two responses, 
# using a standard within-subject t-test:
t_test_result <- visual_metacognition %>%
    dplyr::group_by(Subj_idx,Response) %>%
    dplyr::summarise(Confidence=mean(Confidence), .groups = 'drop_last') %>%
    tidyr::spread(Response, Confidence,sep='.') %>%
    dplyr::mutate(diff=Response.1-Response.0) %>%
    dplyr::pull(diff) %>%
    t.test()

# compare the mean confidence for the two responses, 
# using a non-parametric directional effect test: 
directional_effect_result <- visual_metacognition %>% 
  signcon::test_directional_effect(idv="Subj_idx", 
                                          dv='Confidence', 
                                          iv='Response')

# compare the mean confidence for the two responses, 
# using a non-directional sign-consistency test:
sign_consistency_result <- visual_metacognition %>% 
  signcon::test_sign_consistency(idv="Subj_idx", dv='Confidence', iv='Response')

# compare the mean confidence for the two responses, 
# using a non-directional absolute effect size test:
absolute_es_result <- visual_metacognition %>% 
  signcon::test_absolute_es(idv="Subj_idx", dv='Confidence', iv='Response')

We find that at the group level, the mean difference in confidence between the two responses is 0.02 on a 1-6 scale, and not significantly different from zero ($t(45)=-0.32, p=0.75$). This is also the case when we test for a directional effect with a non-parameteric test ($p=0.63$). In contrast, sign-consistency equals 0.72, significantly above chance ($p<0.001$), indicating that individual subjects were consistently more confident in one response over the other. Similarly, a linear classifier predicted response from confidence with 54% accuracy, significantly above chance ($p<0.001$).

mufcItay/weaknull documentation built on Jan. 29, 2025, 7:28 p.m.