In jepusto/metascore: Model-Based and Robust Score Tests for Meta-Regression

Publication bias and other forms of selective outcome reporting are important threats to the validity of findings from research syntheses---even undermining their special status for informing evidence-based practice and policy guidance. An array of methods have been proposed for detecting selective publication. In particular, Ioannidis and Trikalinos (2007) proposed the Test of Excess Significance (TES), which diagnoses publication bias by comparing the observed number of statistically significant effect sizes to the number expected based on the power of included studies to detect the estimated average effect. Another approach is based on explicit modeling of the selective publication process, as in the weight function model developed by Vevea and Hedges (1995). Under the Vevea-Hedges model, a likelihood ratio test can be used to test for the presence of selective publication. In this note, I demonstrate a connection between these two methods, namely, that the test statistic in TES is the score function of a simple form of the Vevea-Hedges model. This connection motivates a refinement to TES that improves its operating characteristics and allows for between-study heterogeneity through random effects and regression on study characteristics. After describing the refined test, I report simulations evaluating its calibration and power compared to conventional TES, a likelihood ratio test based on the weight function model, and p-uniform.

jepusto/metascore documentation built on June 25, 2019, 8:15 p.m.