orm_validate: Manual validation assistant with Cohen's Kappa

In orisma: Occupational Risk Integrated Systematic Mapping and Analysis

Manual validation assistant with Cohen's Kappa

Description

orm_validate() supports methodological validation of ORISMA's automatic risk extraction by presenting a random sample of classified records for manual review. It then computes Cohen's Kappa to measure agreement between automatic and manual classification.

This addresses a key peer-review concern: distinguishing between "category detected by dictionary" and "risk actually evaluated in study".

The function saves a CSV file pre-filled with automatic classifications that the researcher edits manually, then re-loads for Kappa computation.

Usage

orm_validate(
  mx,
  n_sample = 30L,
  out_dir = "orisma_validation",
  validation_file = NULL,
  seed = 42L,
  lang = getOption("orisma.lang", "en"),
  verbose = getOption("orisma.verbose", TRUE)
)

Arguments

mx	An orisma_matrix object from orm_extract().
n_sample	Integer. Number of records to sample. Default 30.
out_dir	Character. Directory to save validation files.
validation_file	Character or NULL. Path to a completed validation CSV (output of a previous orm_validate() call) for Kappa computation. If NULL, creates the file for manual review.
seed	Integer. Random seed for reproducibility. Default 42.
lang	Character. "en" or "es".
verbose	Logical.

Value

If validation_file is NULL: invisibly returns the path to the validation CSV. If validation_file is provided: returns a data frame with Kappa statistics per category.

orisma documentation built on May 19, 2026, 1:07 a.m.