compute_confusion_matrix_based_scores: Compute confusion matrix-based scores for candidate solution
In langfob/bdpg: Package For Building and Testing Biodiversity Problem Generators

compute_confusion_matrix_based_scores

R Documentation

Compute confusion matrix-based scores for candidate solution

Description

Computes error measures related to confusion matrix, etc. For the purpose of computing a performance score, start by treating the problem as if it's a classification problem, where a selected patch is classified as 1 and an unselected patch is classified as 0. This allows us to use any of the many existing measures developed for classifiers.

Usage

compute_confusion_matrix_based_scores(
  cor_or_app_str,
  num_PUs_in_cand_solution,
  num_PUs,
  num_PUs_in_optimal_solution,
  frac_spp_covered,
  input_err_FP = 0,
  input_err_FN = 0
)

Arguments

`cor_or_app_str`	string
`num_PUs_in_cand_solution`	integer
`num_PUs`	integer number of planning units
`num_PUs_in_optimal_solution`	integer
`frac_spp_covered`	float
`input_err_FP`	float
`input_err_FN`	float

Details

Computations are done with respect to a reference spp occupancy matrix (i.e., COR or APP)

Choice of measures

Doesn't matter too much which measures we use, since this is mostly about demonstrating how to generate and evaluate problems and users will have to choose which measure best aligns with their own goals.

However, it may be that some of these measures are easier to learn to predict than others, so it's good to provide several different ones until we know more. The base case would be to provide the ones that are the simplest and most direct measures over the confusion matrix. Confusion matrix fractions

Computation of confusion matrix elements (TP,TN,FP,FN)

Note that the TP and TN values are computed as the min of the candidate and correct values. This is because the number of "trues" for the candidate can't exceed the number of "trues" in the correct solution by definition.

Similarly, any count of TP or TN that falls short of the corresponding counts in the correct solution represents the number of TP or TN that the candidate got right and using the number of TP or TN from the correct would overstate the candidate's performance.

This all seems a bit odd in the normal classification context because in classification, you would have to know _which_ PUs the classifier got right and count them up. In reserve selection, there could be many ways to get the same final optimal count of PUs in the solution and we don't care _which_ ones are chosen to get that count. However, the same kind of a scoring system can work because we know that anything short of the optimal number represents the existance of False Negatives, i.e., _some_ PUs who should have been included. Similarly, any count greater than the optimal count implies the existance of False Positives, i.e., _some_ PUs who should NOT have been included.

Since nearly all classification performance scores are based on some combination of the 4 values from the confusion matrix (TP,TN,FP,FN), choosing those 4 values sets us up to compute all these scores.

Source of formulas for measures

I got nearly all of these measures from one paper a while ago and I can't remember exactly what paper it was at the moment. Need to look this up again. I think it might have been the 2011 Liu et al. paper in the references section below.