rater_network_analysis: Analyze rater agreement, disagreement, and severity-direction...

In mfrmr: Estimation and Diagnostics for Many-Facet Measurement Models

Analyze rater agreement, disagreement, and severity-direction networks

Description

Analyze rater agreement, disagreement, and severity-direction networks

Usage

rater_network_analysis(
  fit,
  diagnostics = NULL,
  rater_facet = NULL,
  context_facets = NULL,
  mode = c("agreement", "disagreement", "severity_direction"),
  weight_metric = NULL,
  min_pair_n = 1,
  min_weight = 0,
  score_diff_tolerance = 0,
  severity_continuity = 0.5,
  exact_warn = 0.5,
  corr_warn = 0.3,
  include_graph = FALSE
)

Arguments

fit	Output from fit_mfrm().
diagnostics	Optional output from diagnose_mfrm().
rater_facet	Name of the rater-like facet. If omitted, mfrmr uses the same heuristic as interrater_agreement_table().
context_facets	Facets defining shared scoring contexts. By default, the person facet and all non-rater facets are used.
mode	Network definition. "agreement" builds an undirected network whose edge weights represent observed agreement. "disagreement" builds an undirected network whose edge weights represent observed disagreement. "severity_direction" builds a directed network: an edge from rater A to rater B means A assigned higher scores than B in shared contexts and is therefore relatively more lenient under the usual higher-score-is-better rating convention.
weight_metric	Pair-level weight used for "agreement" or "disagreement" networks. Defaults to Exact for agreement and MAD for disagreement. Available pair columns include Exact, Adjacent, Corr, MAD, OneMinusExact, and AbsMeanDiff.
min_pair_n	Minimum number of shared contexts required for a rater pair to contribute an edge.
min_weight	Minimum edge weight retained in the graph.
score_diff_tolerance	Score-difference tolerance for directed severity networks. With the default 0, any higher score contributes to the outgoing leniency edge. Larger values reproduce thresholded disagreement displays such as "only differences greater than 3 marks".
severity_continuity	Continuity constant added to incoming and outgoing strengths before computing the finite severity index -log((OutStrength + c) / (InStrength + c)).
exact_warn, corr_warn	Passed to interrater_agreement_table() to keep pair flags consistent with the tabular agreement view.
include_graph	If TRUE, include the underlying igraph object in the returned bundle.

Details

This function implements a package-native rater-effect network view complementary to MFRM output. It follows the pairwise-network logic used in Lamprianou's rater-effect network work: nodes are raters, edges summarize pairwise relationships among raters in shared scoring contexts, and directed disagreement edges can be interpreted as relative leniency/severity indicators. These network summaries are descriptive diagnostics, not Rasch logit estimates and not formal fit statistics.

For mode = "severity_direction", outgoing strength means the rater more often assigned higher scores than comparison raters; incoming strength means comparison raters more often assigned higher scores than this rater. The reported SeverityIndex is positive for relatively severe raters and negative for relatively lenient raters, but it is on a network-analysis scale and should not be read as an MFRM severity logit.

Value

A bundle of class mfrm_rater_network containing:

summary

One-row graph summary.

node_metrics

Rater-level degree, strength, centrality, and severity-direction summaries.

edge_metrics

Retained rater-pair network edges.

pair_metrics

All eligible pairwise agreement and directional comparison metrics before edge thresholding.

caveats

Interpretation notes and sparse-design warnings.

source_interrater

The underlying interrater_agreement_table() output used for agreement statistics.

References

Lamprianou, I. (2018). Investigation of rater effects using Social Network Analysis and Exponential Random Graph Models. Educational and Psychological Measurement, 78(3), 430-459.

Lamprianou, I. (2025). Network Analysis for the investigation of rater effects in language assessment: A comparison of ChatGPT vs human raters. Research Methods in Applied Linguistics, 4, 100205.

See Also

interrater_agreement_table(), plot_interrater_agreement(), mfrm_network_analysis(), plot.mfrm_bundle()

Examples

toy <- load_mfrmr_data("example_core")
fit <- fit_mfrm(toy, "Person", c("Rater", "Criterion"), "Score",
                method = "JML", maxit = 30)
if (requireNamespace("igraph", quietly = TRUE)) {
  rn <- rater_network_analysis(fit, mode = "severity_direction")
  rn$summary
  head(rn$node_metrics)
  plot(rn, type = "severity", draw = FALSE)
}

mfrmr documentation built on June 13, 2026, 1:07 a.m.