estimate_rank_double_posterior: Signal subspace dimension estimation in high-dimensional...

In WenlanzZ/MKDim: Determine Signal Space Dimension

Signal subspace dimension estimation in high-dimensional matrix by double posterior.

Description

Estimate the dimension of a signal-rich subspace in large, high-dimensional data.

Usage

estimate_rank_double_posterior(s, p, verbose, ...)

Arguments

s	a subspace class.
p	threshold for selecting changepoint. Default is 0.90.
verbose	output message
...	Extra parameters

Value

Returns a list with entries:

ndf:

The number of degrees of freedom of x.

pdim:

The number of dimensions of x.

components:

A series of right singular vectors estimated.

var_correct:

Corrected population variance for Marcenko-Pastur distribution.

transpose_flag:

A logical value indicating whether the matrix x is transposed.

irl:

A data frame of scaled eigenvalues for specified components and corresponding dimensions.

sigma_a:

A vector of corrected eigenvalues up to max(components).

mp_irl:

A data frame of sampled expected eigenvalues from Marcenko-Pastur for specified components and corresponding dimensions.

sigma_mp:

A vector of samped expected eigenvalues from Marcenko-Pastur up to max(components).

v:

Right singular vectors of x matrix for specified components.

u:

Left singular vectors of x matrix or specified components.

dimension:

Estimated signal subspace dimension.

bcp_irl:

Probability of change in mean and posterior means of eigenvalue difference between $x$ and $N$.

bcp_post:

Probability of change in mean and posterior means of bcp_rirl.

Details

We estimate the intrinsic dimension of a signal-rich subspace in large high-dimensional data by decomposing matrix into a signal-plus-noise space and approximate the signal-rich subspace with a rank K approximation \hat{x}=∑_{k=1}^{K}d_ku_k{v_k}^T. To estimate rank K, we propose a simple procedure assuming that matrix x is composed of a low-rank signal matrix S and an average general noise random matrix \bar{N}. It has been shown that the average eigenvalues of random matrices N follows a universal Marchenko-Pastur (MP) distribution. We hypothesize that the deviation of eigenvalues of x from the MP distribution indicates the intrinsic dimension of signal-rich subspace.

See Also

[RMTstat] for details of Marchenko-Pastur distribution.

https://dracodoc.wordpress.com/2014/07/21/ a-simple-algorithm-to-detect-flat-segments-in-noisy-signals/ for detection of flat and spike in noisy signals

Examples

x <- x_sim(n = 100, p = 150, ncc = 10, var = c(rep(10, 5), rep(1, 5)))
results <- x %>%
create_subspace(components = 8:30) %>%
correct_eigenvalues() %>%
estimate_rank_double_posterior()

str(results)
plot(results$subspace, changepoint = results$dimension,
     annotation = 10)
modified_legacyplot(results$bcp_irl, annotation = 10)
modified_legacyplot(results$bcp_post, annotation = 10)

WenlanzZ/MKDim documentation built on July 30, 2022, 7:25 a.m.