Anomaly Detection in High Dimensional Data Space
This package is a modification of HDoutliers package. The HDoutliers algorithm is a powerful unsupervised algorithm for detecting anomalies in high-dimensional data, with a strong theoretical foundation. However, it suffers from some limitations that significantly hinder its performance level, under certain circumstances. In this package, we propose an algorithm that addresses these limitations. We define an anomaly as an observation that deviates markedly from the majority with a large distance gap. An approach based on extreme value theory is used for the anomalous threshold calculation.
A companion paper to this work is available here. Using various synthetic and real datasets, we demonstrate the wide applicability and usefulness of our algorithm, which we call the stray algorithm. We also demonstrate how this algorithm can assist in detecting anomalies present in other data structures using feature engineering. We show the situations where the stray algorithm outperforms the HDoutliers algorithm both in accuracy and computational time.
This package is still under development and this repository contains a development version of the R package stray.
You can install oddstream from github with:
# install.packages("devtools") devtools::install_github("pridiltal/stray")
library(stray) require(ggplot2) #> Loading required package: ggplot2 set.seed(1234) data <- c(rnorm(1000, mean = -6), 0, rnorm(1000, mean = 6)) outliers <- find_HDoutliers(data, knnsearchtype = "brute") names(outliers) #>  "outliers" "out_scores" "type" display_HDoutliers(data, outliers)
set.seed(1234) n <- 1000 # number of observations nout <- 10 # number of outliers typical_data <- matrix(rnorm(2*n), ncol = 2, byrow = TRUE) out <- matrix(5*runif(2*nout,min=-5,max=5), ncol = 2, byrow = TRUE) data <- rbind(out, typical_data ) outliers <- find_HDoutliers(data, knnsearchtype = "brute") display_HDoutliers(data, outliers)
For data with more than two dimensions, two dimensional scatterplot is produced using the first two pricipal components.
data <- rbind(matrix(rnorm(144), ncol = 3), c(10,12,10),c(3,7,10)) output <- find_HDoutliers(data, knnsearchtype = "brute") display_HDoutliers(data, out = output)
More examples are available from our paper Anomaly Detection in High Dimensional Data
outliers<-find_HDoutliers(data_c[,1:2], knnsearchtype= "brute") p <- display_HDoutliers(data_c[,1:2], outliers)+ ggplot2::ggtitle("data_c") print(p)
outliers<-find_HDoutliers(data_d[,1:2], knnsearchtype= "brute") p <- display_HDoutliers(data_d[,1:2], outliers)+ ggplot2::ggtitle("data_d") print(p)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.