# Scoring_system: The main function for algorithms scoring based on accuracy,... In bishun945/FCMm: Fuzzy Cluster Method Based on the Optimized m Value (Fuzzifier)

## Description

The main function for algorithms scoring based on accuracy, precision, and effectiveness.

## Usage

 ``` 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24``` ```Scoring_system( Inputs, method = "sort-based", param_sort = list(decreasing = TRUE, max.score = NULL), param_interval = list(trim = FALSE, reward.punishment = TRUE, decreasing = TRUE, hundred.percent = FALSE), remove.negative = FALSE, accuracy.metrics = c("MAE", "CMAPE"), precision.metrics = c("BIAS", "CMRPE") ) Scoring_system_bootstrap( Times = 1000, Inputs, replace = TRUE, method = "sort-based", metrics_used = 1, param_sort = list(decreasing = TRUE, max.score = NULL), param_interval = list(trim = FALSE, reward.punishment = TRUE, decreasing = TRUE, hundred.percent = FALSE), remove.negative = FALSE, dont_blend = FALSE, verbose = TRUE ) ```

## Arguments

 `Inputs` The list returned form function Getting_Asses_results `method` The method selected to score algorithms: `sort-based` (default) which is scored by the sort of accuracy and precision metrics (see more in Score_algorithms_sort). `sort-based2` which is scored by the sort of accuracy and precision metrics (see more in Score_algorithms_sort). `interval-based` which is relatively scored by the interval of accuracy and precision (used by Brewin et al. (2015) and Neil et al. (2019)). See more in Score_algorithms_interval). `param_sort` The parameters of function Score_algorithms_sort `param_interval` The parameters of function Score_algorithms_interval `remove.negative` Option to replace the negative score as zero (default as `FALSE`) `accuracy.metrics` accuracy used metrics, default as `c("MAE", "CMAPE")` `precision.metrics` precision used metrics, default as `c("BIAS", "CMRPE")` `Times` Parameter of `Scoring_system_bootstrap`. The bootstrap time for running `Scoring_system` (default as `1000`) `replace` Parameter of `Scoring_system_bootstrap`. The sample method for bootstrap running. Default as `TRUE`. See more details in sample. `metrics_used` The metric combination used in the function. Default is `1`. If `metrics_used = 1` then the used metrics are `c("MAE", "CMAPE", "BIAS", "CMRPE")` If `metrics_used = 2` (dont use this) then the used metrics are `c("MAE", "CMAPE", "BIAS", "CMRPE", "RATIO")` `dont_blend` Whether to runing the algorithm blending process. Default is `FALSE`. This is useful when you just want to score the candidate algorithms. `verbose` Show the iteration message.

## Details

The `Accuracy` and `Precision` is newly defined in `FCMm` package (referred by Hooker et al. (2005)):

• `Accuracy` is the estimation of how close the result of the experiment is to the true value.

• `Precision` is the estimation of how excatly the result is determined independently of any true value.

In other words, `Accuracy` is telling a story truthfully and precision is how similarly the story is represented over and over again. Here we use AE, a vector for each sample, for instance:

• `Accuracy` is the aggregation (no matter mean or median, in fuzzy calculation process), we use mean to some extent.

• `Precision` is actually the stability of AE (reproducebility) which means the error produced by the algorithm is under certain control.

Finally, the function will multiply the total score (`Accuracy` + `Precision`) by the effectiveness (i.e., Valid_percent returned by Assessment_via_cluster).

## Value

The result of `Scoring_system` are including:

• Total_score Data.frame of final score result with algorithm as column and cluster as row.

• Accuracy Data.frame of `Accuracy` score with algorithm as column and cluster as row.

• Precision Data.frame of `Precision` score with algorithm as column and cluster as row.

• Effectiveness Data.frame of `Effectiveness` score with algorithm as column and cluster as row.

• Accuracy_list List including data.frames of used `Accuracy` metrics.

• Precision_list List including data.frames of used `Precision` metrics.

• Total_score.melt Melted data.frame of Total_score for plotting.

• Opt_algorithm The optimal algorithm names for each cluster.

• Inputs Inputs of this function.

The result of `Scoring_system_bootstrap` are including:

• Times The times of bootstrap running.

• Score_all_clusters The total score for algorithms across all clusters.

• Score_list All times of bootstrapping results are recorded in it.

• Score_list_melt Melted `Score_list`.

• Opt_algorithm_list The optimal algorithm for every runing.

• Opt_algorithm The optimal algorithm defined by mode of `Opt_algorithm_list` for each cluster.

• Remove_algorithm The algorithms to be removed when blending.

• plot_col The col plot of `Score_list_melt`.

• plot_scatter The scatter plot of measured and predicted Chla concentration colored by clusters.

• plot_scatter_opt The scatter plot of measured and predicted Chla concentration colored by clusters for optimized algorithms.

• Blend_result The results from the inherent function `Chla_algorithms_blend`.

• dt_Chla Data.frame with combination of candidate algortihms and blended results.

• Chla_blend The blended Chla concentration by score results.

• Results_of_scoring_system A list including all results of Scoring_system function.

• metric_results A result of Assessment_via_cluster which includes the Chla blend results.

## Note

`Scoring_system_bootstrap` is the bootstrap mode of `Scoring_system` which is useful when the outcome is unstable for large number of samples. The default boostrap time in `Scoring_system_bootstrap` is set as `1000` and the result of it is the list of several aggregated data.frames and standard deviations.

## References

• Hooker S B. Second SeaWiFS HPLC analysis round-robin experiment (SeaHARRE-2)[M]. National Aeronautics and Space Administration, Goddard Space Flight Center, 2005.

• Seegers B N, Stumpf R P, Schaeffer B A, et al. Performance metrics for the assessment of satellite data products: an ocean color case study[J]. Optics express, 2018, 26(6): 7404-7422.

• Neil C, Spyrakos E, Hunter P D, et al. A global approach for chlorophyll-a retrieval across optically complex inland waters based on optical water types[J]. Remote Sensing of Environment, 2019, 229: 159-178.

• Brewin R J W, Sathyendranath S, Müller D, et al. The Ocean Colour Climate Change Initiative: III. A round-robin comparison on in-water bio-optical algorithms[J]. Remote Sensing of Environment, 2015, 162: 271-294.

• Moore T S, Dowell M D, Bradt S, et al. An optical water type framework for selecting and blending retrievals from bio-optical algorithms in lakes and coastal waters[J]. Remote sensing of environment, 2014, 143: 97-111.

Other Algorithm assessment: `Assessment_via_cluster()`, `Getting_Asses_results()`, `Sampling_via_cluster()`, `Score_algorithms_interval()`, `Score_algorithms_sort()`
 ``` 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55``` ```library(FCMm) library(ggplot2) library(magrittr) library(stringr) data("Nechad2015") x <- Nechad2015[,3:11] wv <- gsub("X","",names(x)) %>% as.numeric set.seed(1234) w <- sample.int(nrow(x)) x <- x[w, ] names(x) <- wv nb = 4 # Obtained from the vignette "Cluster a new dataset by FCMm" set.seed(1234) FD <- FuzzifierDetermination(x, wv, do.stand=TRUE) result <- FCM.new(FD, nb, fast.mode = TRUE) p.spec <- plot_spec(result, show.stand=TRUE) print(p.spec\$p.cluster.spec) Chla <- Nechad2015\$X.Chl_a..ug.L.[w] Chla[Chla >= 999] <- NA dt_Chla <- run_all_Chla_algorithms(x) %>% as.data.frame dt_Chla <- data.frame(Chla_true = Chla, BR_Gil10 = dt_Chla\$BR_Gil10, OC4_OLCI = dt_Chla\$OC4_OLCI, OCI_Hu12 = dt_Chla\$OCI_Hu12, NDCI_Mi12= dt_Chla\$NDCI_Mi12) %>% round(3) w = which(!is.na(dt_Chla\$Chla_true)) dt_Chla = dt_Chla[w,] memb = result\$res.FCM\$u[w,] %>% round(4) cluster = result\$res.FCM\$cluster[w] Asses_results <- Getting_Asses_results(sample.size=length(cluster), pred = dt_Chla[,-1], meas = data.frame(dt_Chla[,1]), memb = memb, cluster = cluster) Score = Scoring_system(Asses_results) # show the total score table knitr::kable(round(Score\$Total_score, 2)) # Examples of `Scoring_system_bootstrap` set.seed(1234) Score_boo <- Scoring_system_bootstrap(Times = 3, Asses_results) # try to set large `Times` when using your own data # Show the bar plot of scores Score_boo\$plot_col # Show the scatter plot of measure-estimation pairs Score_boo\$plot_scatter # Show error metrics knitr::kable(round(Score_boo\$metric_results\$MAE, 2), caption = "MAE") knitr::kable(round(Score_boo\$metric_results\$CMAPE, 2), caption = "CAPE") knitr::kable(round(Score_boo\$metric_results\$BIAS, 2), caption = "BIAS") knitr::kable(round(Score_boo\$metric_results\$CMRPE, 2), caption = "CRPE") # you would see the blending estimations outperform than other candidates ```