alg_compare: Comparison of the performances of the optimization algorithms...

In sowmyachandrasekaran17/algCompare: A Robust Statistical Framework for Comparing Optimization Heuristics

Description

The alg_compare function performs an upper tail hypothesis testing to identify if Alg_new outperforms Alg_bm. The important feature of this function is the severity measure, a key statistic that stringently validates the performance in presence of small discrepancies. This discrepancy accounts for the practically insignificant improvement, which could have occurred due to several factors such as computer accuracy (i.e., floating points), variable types (4-byte float, 8-byte float, 10-byte float), or even the stopping criteria that is the error threshold when the algorithms are stopped.Secondly, discrepancy can be considered as the improvement that is practically insignificant, which is user-defined and depends on the application being tested.

The severity support for the decision of not-reject or reject is obtained for the chosen discrepancy range. The normalized area under the severity discrepancy curve is proposed as a new measure, which summarizes the support of severity over the discrepancy region of interest.The value of the normalized area under the severity discrepancy curve is closer to 1 then it conveys stronger support for the decision made by the hypothesis testing over the discrepancy region of interest.A value closer to 0 signifies no support.

Additionally, the severity measure and power is plotted for the chosen discrepancy range in a single plot.

Usage

alg_compare(
  Alg_bm,
  Alg_new,
  alpha = 0.05,
  discrepancy_range = 0.001,
  bootstrapping = FALSE,
  measure = "mean",
  nsample = 10000
)

Arguments

Alg_bm	an n-dimensional vector of the achieved objective values obtained for n runs by the benchmark algorithm
Alg_new	an n-dimensional vector of the achieved objective values obtained for n runs by the new algorithm
alpha	is the significance level (0< alpha <1, default is 0.05)
discrepancy_range	is the range of practically insignificant improvement (application dependent, a user_defined_value, default is .001)
bootstrapping	indicates if bootstrapping needs to be performed with hypothesis testing, default "FALSE", accepts "TRUE" or "FALSE", choose "FALSE" for normal data-set and "TRUE" in-case of non-normal data-set
measure	indicates which measure to be compared among algorithms in the hypothesis testing, can take "mean", "median", "mode".
nsample	the number of bootstrap samples, if needed

Details

The statement Alg_new outperforms Alg_bm is equivalent to Alg_bm > Alg_new, which can be formulated as the statistical hypothesis H0: Alg_bm-Alg_new <= 0. This hypothesis H0 will be tested against the alternate hypothesis Ha: Alg_bm-Alg_new > 0, which states that Alg_new is better than Alg_bm for a minimization problem.

• Null hypothesis H0: Alg_new does not outperform Alg_bm

• Alternate hypothesis Ha: Alg_new outperforms Alg_bm

Value

returns a list of performance metrics

• decision - H0 either as not-reject or rejected based on the p-value

• p - the p-value of the test

• sev_notRejectH0 - a numeric vector of severity measure for not-rejecting the null hypothesis for the range of discrepancies

• sev_rejectH0 - a numeric vector of severity measure for rejecting the null hypothesis for the range of discrepancies

• discrepancy - the range of user defined discrepancy

• power - a numeric vector comprising the power function corresponding to each discrepancy

• sev_auc - Normalized area under the severity discrepancy curve

Author(s)

Sowmya Chandrasekaran

Examples

## Prerequisite 1: Create two vectors A and B. Let A and B be the optimum achieved.
set.seed(123)
A <-abs(rnorm(20,mean=.1,sd=1))
set.seed(123)
B <- abs(rnorm(20,mean=.01,sd=.1))

## Example 1: Compare A and B to check if B outperforms A.
## H0: B does not outperform A vs Ha: B outperforms A
result_example1 <- alg_compare(A,B,discrepancy_range=.001,bootstrapping=FALSE,measure="mean")

## Interpretation of the result: The decision based on the p-value is to reject H0.
## The severity value always remains close to 1, suggesting that it highly
## supports the decision over the complete range of discrepancy. The value of sev_auc is close to 1 signifying strong support for the decision made by the hypothesis test.

## Example 2: Compare A and B to check if A outperforms B.
##H0: A does not outperform B vs Ha: A outperforms B
result_example2 <- alg_compare(B,A,bootstrapping=FALSE,measure="mean")

## Interpretation of the result: The decision based on the p-value is to not-reject H0.
## The severity value always remains close to 1, suggesting that it highly supports
## the decision over the complete range of discrepancy. The value of sev_auc is close to 1 signifying strong support for the decision made by the hypothesis test.

## Prerequisite 2: Create sphere function and optimize it with Nelder-Mead and SANN for 10 runs
## Sphere function
sphere <- function(xx)
{
sum <- sum(xx^2)
y <- sum
return(y)
}
 ## Initialization of variables
nm_optim <- NULL
sann_optim <-NULL
runs <- 10
set.seed(123)
## Optimizing Sphere function with NM and SANN
for(i in 1:runs){
 start <- (runif(2,min=(-5),max=(5)))# random initial start at each run
 nm <-optim(start, sphere, method = "Nelder-Mead")
 nm_mean <- (nm$value)
 nm_optim <- c(nm_optim,nm_mean)# Complete Results for NM Algorithm
 sann <- optim(start, sphere, method = "SANN")
 sann_mean <- (sann$value)
 sann_optim <- c(sann_optim,sann_mean)# Complete Results for SANN Algorithm
}

## Example 3:
## Compare Nelder-Mead and SANN to check if NM outperforms SANN
## Alg_bm <- SANN, Alg_new <- NM, Tests if NM outperforms SANN
## ( Null hypothesis H0: Alg_new does not outperform Alg_bm
## Alt hypothesis Ha: Alg_new outperforms Alg_bm)
result_nm_outperforms_sann <- alg_compare(sann_optim,nm_optim,bootstrapping=FALSE,measure="mean")

 ## Interpretation of the result: The decision based on the p-value is to reject H0.
## The severity value is high until a discrepancy value of 10^-4
## after which it decreases to zero. This signifies that NM outperforms
## SANN only with a very small variation that is less than 10^-4. The value of sev_auc is very less signifying no support for the decision made by the hypothesis test.

## Example 4:
## Compare Nelder-Mead and SANN to check if SANN outperforms NM
 result_sann_outperforms_nm <- alg_compare(nm_optim,sann_optim,bootstrapping=FALSE,measure="mean")

 ## Interpretation of the result: The decision based on the p-value is to not-reject H0.
 ##The severity value always remains close to 1, suggesting that it highly supports
 ## the decision over the complete range of discrepancy. The value of sev_auc is close to 1 signifying strong support for the decision made by the hypothesis test.

sowmyachandrasekaran17/algCompare documentation built on Dec. 23, 2021, 4:24 a.m.