Scoring_system: The main function for algorithms scoring based on accuracy,...
In bishun945/FCMm: Fuzzy Cluster Method Based on the Optimized m Value (Fuzzifier)

Description Usage Arguments Details Value Note References See Also Examples

The main function for algorithms scoring based on accuracy, precision, and effectiveness.

Scoring_system(
  Inputs,
  method = "sort-based",
  param_sort = list(decreasing = TRUE, max.score = NULL),
  param_interval = list(trim = FALSE, reward.punishment = TRUE, decreasing = TRUE,
    hundred.percent = FALSE),
  remove.negative = FALSE,
  accuracy.metrics = c("MAE", "CMAPE"),
  precision.metrics = c("BIAS", "CMRPE")
)

Scoring_system_bootstrap(
  Times = 1000,
  Inputs,
  replace = TRUE,
  method = "sort-based",
  metrics_used = 1,
  param_sort = list(decreasing = TRUE, max.score = NULL),
  param_interval = list(trim = FALSE, reward.punishment = TRUE, decreasing = TRUE,
    hundred.percent = FALSE),
  remove.negative = FALSE,
  dont_blend = FALSE,
  verbose = TRUE
)

`Inputs`	The list returned form function Getting_Asses_results
`method`	The method selected to score algorithms: `sort-based` (default) which is scored by the sort of accuracy and precision metrics (see more in Score_algorithms_sort). `sort-based2` which is scored by the sort of accuracy and precision metrics (see more in Score_algorithms_sort). `interval-based` which is relatively scored by the interval of accuracy and precision (used by Brewin et al. (2015) and Neil et al. (2019)). See more in Score_algorithms_interval).
`param_sort`	The parameters of function Score_algorithms_sort
`param_interval`	The parameters of function Score_algorithms_interval
`remove.negative`	Option to replace the negative score as zero (default as `FALSE`)
`accuracy.metrics`	accuracy used metrics, default as `c("MAE", "CMAPE")`
`precision.metrics`	precision used metrics, default as `c("BIAS", "CMRPE")`
`Times`	Parameter of `Scoring_system_bootstrap`. The bootstrap time for running `Scoring_system` (default as `1000`)
`replace`	Parameter of `Scoring_system_bootstrap`. The sample method for bootstrap running. Default as `TRUE`. See more details in sample.
`metrics_used`	The metric combination used in the function. Default is `1`. If `metrics_used = 1` then the used metrics are `c("MAE", "CMAPE", "BIAS", "CMRPE")` If `metrics_used = 2` (dont use this) then the used metrics are `c("MAE", "CMAPE", "BIAS", "CMRPE", "RATIO")`
`dont_blend`	Whether to runing the algorithm blending process. Default is `FALSE`. This is useful when you just want to score the candidate algorithms.
`verbose`	Show the iteration message.

The Accuracy and Precision is newly defined in FCMm package (referred by Hooker et al. (2005)):

Accuracy is the estimation of how close the result of the experiment is to the true value.
Precision is the estimation of how excatly the result is determined independently of any true value.

In other words, Accuracy is telling a story truthfully and precision is how similarly the story is represented over and over again. Here we use AE, a vector for each sample, for instance:

Accuracy is the aggregation (no matter mean or median, in fuzzy calculation process), we use mean to some extent.
Precision is actually the stability of AE (reproducebility) which means the error produced by the algorithm is under certain control.

Finally, the function will multiply the total score (Accuracy + Precision) by the effectiveness (i.e., Valid_percent returned by Assessment_via_cluster).

The result of Scoring_system are including:

Total_score Data.frame of final score result with algorithm as column and cluster as row.
Accuracy Data.frame of Accuracy score with algorithm as column and cluster as row.
Precision Data.frame of Precision score with algorithm as column and cluster as row.
Effectiveness Data.frame of Effectiveness score with algorithm as column and cluster as row.
Accuracy_list List including data.frames of used Accuracy metrics.
Precision_list List including data.frames of used Precision metrics.
Total_score.melt Melted data.frame of Total_score for plotting.
Opt_algorithm The optimal algorithm names for each cluster.
Inputs Inputs of this function.

The result of Scoring_system_bootstrap are including:

Times The times of bootstrap running.
Score_all_clusters The total score for algorithms across all clusters.
Score_list All times of bootstrapping results are recorded in it.
Score_list_melt Melted Score_list.
Opt_algorithm_list The optimal algorithm for every runing.
Opt_algorithm The optimal algorithm defined by mode of Opt_algorithm_list for each cluster.
Remove_algorithm The algorithms to be removed when blending.
plot_col The col plot of Score_list_melt.
plot_scatter The scatter plot of measured and predicted Chla concentration colored by clusters.
plot_scatter_opt The scatter plot of measured and predicted Chla concentration colored by clusters for optimized algorithms.
Blend_result The results from the inherent function Chla_algorithms_blend.
dt_Chla Data.frame with combination of candidate algortihms and blended results.
Chla_blend The blended Chla concentration by score results.
Results_of_scoring_system A list including all results of Scoring_system function.
metric_results A result of Assessment_via_cluster which includes the Chla blend results.

Scoring_system_bootstrap is the bootstrap mode of Scoring_system which is useful when the outcome is unstable for large number of samples. The default boostrap time in Scoring_system_bootstrap is set as 1000 and the result of it is the list of several aggregated data.frames and standard deviations.

Hooker S B. Second SeaWiFS HPLC analysis round-robin experiment (SeaHARRE-2)[M]. National Aeronautics and Space Administration, Goddard Space Flight Center, 2005.
Seegers B N, Stumpf R P, Schaeffer B A, et al. Performance metrics for the assessment of satellite data products: an ocean color case study[J]. Optics express, 2018, 26(6): 7404-7422.
Neil C, Spyrakos E, Hunter P D, et al. A global approach for chlorophyll-a retrieval across optically complex inland waters based on optical water types[J]. Remote Sensing of Environment, 2019, 229: 159-178.
Brewin R J W, Sathyendranath S, Müller D, et al. The Ocean Colour Climate Change Initiative: III. A round-robin comparison on in-water bio-optical algorithms[J]. Remote Sensing of Environment, 2015, 162: 271-294.
Moore T S, Dowell M D, Bradt S, et al. An optical water type framework for selecting and blending retrievals from bio-optical algorithms in lakes and coastal waters[J]. Remote sensing of environment, 2014, 143: 97-111.

Other Algorithm assessment: Assessment_via_cluster(), Getting_Asses_results(), Sampling_via_cluster(), Score_algorithms_interval(), Score_algorithms_sort()

library(FCMm) 
library(ggplot2) 
library(magrittr)
library(stringr)
data("Nechad2015")
x <- Nechad2015[,3:11]
wv <- gsub("X","",names(x)) %>% as.numeric
set.seed(1234)
w <- sample.int(nrow(x))
x <- x[w, ]
names(x) <- wv
nb = 4 # Obtained from the vignette "Cluster a new dataset by FCMm"
set.seed(1234)
FD <- FuzzifierDetermination(x, wv, do.stand=TRUE)
result <- FCM.new(FD, nb, fast.mode = TRUE)
p.spec <- plot_spec(result, show.stand=TRUE)
print(p.spec$p.cluster.spec)
Chla <- Nechad2015$X.Chl_a..ug.L.[w]
Chla[Chla >= 999] <- NA
dt_Chla <- run_all_Chla_algorithms(x) %>% as.data.frame
dt_Chla <- data.frame(Chla_true = Chla,
BR_Gil10 = dt_Chla$BR_Gil10, 
OC4_OLCI = dt_Chla$OC4_OLCI, 
OCI_Hu12 = dt_Chla$OCI_Hu12, 
NDCI_Mi12= dt_Chla$NDCI_Mi12) %>% round(3)
w = which(!is.na(dt_Chla$Chla_true))
dt_Chla = dt_Chla[w,]
memb = result$res.FCM$u[w,] %>% round(4)
cluster =  result$res.FCM$cluster[w]
Asses_results <- Getting_Asses_results(sample.size=length(cluster), 
pred = dt_Chla[,-1], meas = data.frame(dt_Chla[,1]), memb = memb, 
cluster = cluster)
Score = Scoring_system(Asses_results)
# show the total score table
knitr::kable(round(Score$Total_score, 2))

# Examples of `Scoring_system_bootstrap`

set.seed(1234)
Score_boo <- Scoring_system_bootstrap(Times = 3, Asses_results) 
# try to set large `Times` when using your own data

# Show the bar plot of scores
Score_boo$plot_col

# Show the scatter plot of measure-estimation pairs
Score_boo$plot_scatter

# Show error metrics
knitr::kable(round(Score_boo$metric_results$MAE, 2), caption = "MAE")
knitr::kable(round(Score_boo$metric_results$CMAPE, 2), caption = "CAPE")
knitr::kable(round(Score_boo$metric_results$BIAS, 2), caption = "BIAS")
knitr::kable(round(Score_boo$metric_results$CMRPE, 2), caption = "CRPE")

# you would see the blending estimations outperform than other candidates