The main function for algorithms scoring based on accuracy, precision, and effectiveness.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Scoring_system( Inputs, method = "sort-based", param_sort = list(decreasing = TRUE, max.score = NULL), param_interval = list(trim = FALSE, reward.punishment = TRUE, decreasing = TRUE, hundred.percent = FALSE), remove.negative = FALSE, accuracy.metrics = c("MAE", "CMAPE"), precision.metrics = c("BIAS", "CMRPE") ) Scoring_system_bootstrap( Times = 1000, Inputs, replace = TRUE, method = "sort-based", metrics_used = 1, param_sort = list(decreasing = TRUE, max.score = NULL), param_interval = list(trim = FALSE, reward.punishment = TRUE, decreasing = TRUE, hundred.percent = FALSE), remove.negative = FALSE, dont_blend = FALSE, verbose = TRUE )
The list returned form function Getting_Asses_results
The method selected to score algorithms:
The parameters of function Score_algorithms_sort
The parameters of function Score_algorithms_interval
Option to replace the negative score as zero (default as
accuracy used metrics, default as
precision used metrics, default as
The metric combination used in the function. Default is
Whether to runing the algorithm blending process. Default is
Show the iteration message.
Precision is newly defined in
FCMm package (referred by
Hooker et al. (2005)):
Accuracy is the estimation of how close the result of the experiment is to the
Precision is the estimation of how excatly the result is determined independently
of any true value.
In other words,
Accuracy is telling a story truthfully and precision is how similarly
the story is represented over and over again.
Here we use AE, a vector for each sample, for instance:
Accuracy is the aggregation (no matter mean or median, in fuzzy calculation process),
we use mean to some extent.
Precision is actually the stability of AE (reproducebility) which means the error
produced by the algorithm is under certain control.
Finally, the function will multiply the total score (
Precision) by the
effectiveness (i.e., Valid_percent returned by Assessment_via_cluster).
The result of
Scoring_system are including:
Total_score Data.frame of final score result with algorithm as column and cluster as row.
Accuracy Data.frame of
Accuracy score with algorithm as column and cluster as row.
Precision Data.frame of
Precision score with algorithm as column and cluster as row.
Effectiveness Data.frame of
Effectiveness score with algorithm as column and cluster as row.
Accuracy_list List including data.frames of used
Precision_list List including data.frames of used
Total_score.melt Melted data.frame of Total_score for plotting.
Opt_algorithm The optimal algorithm names for each cluster.
Inputs Inputs of this function.
The result of
Scoring_system_bootstrap are including:
Times The times of bootstrap running.
Score_all_clusters The total score for algorithms across all clusters.
Score_list All times of bootstrapping results are recorded in it.
Opt_algorithm_list The optimal algorithm for every runing.
Opt_algorithm The optimal algorithm defined by mode of
for each cluster.
Remove_algorithm The algorithms to be removed when blending.
plot_col The col plot of
plot_scatter The scatter plot of measured and predicted Chla concentration colored by clusters.
plot_scatter_opt The scatter plot of measured and predicted Chla concentration colored by clusters for optimized algorithms.
Blend_result The results from the inherent function
dt_Chla Data.frame with combination of candidate algortihms and blended results.
Chla_blend The blended Chla concentration by score results.
Results_of_scoring_system A list including all results of Scoring_system function.
metric_results A result of Assessment_via_cluster which includes the Chla blend results.
Scoring_system_bootstrap is the bootstrap mode of
Scoring_system which is useful when
the outcome is unstable for large number of samples. The default boostrap time in
is set as
1000 and the result of it is the list of several aggregated data.frames and standard deviations.
Hooker S B. Second SeaWiFS HPLC analysis round-robin experiment (SeaHARRE-2)[M]. National Aeronautics and Space Administration, Goddard Space Flight Center, 2005.
Seegers B N, Stumpf R P, Schaeffer B A, et al. Performance metrics for the assessment of satellite data products: an ocean color case study[J]. Optics express, 2018, 26(6): 7404-7422.
Neil C, Spyrakos E, Hunter P D, et al. A global approach for chlorophyll-a retrieval across optically complex inland waters based on optical water types[J]. Remote Sensing of Environment, 2019, 229: 159-178.
Brewin R J W, Sathyendranath S, Müller D, et al. The Ocean Colour Climate Change Initiative: III. A round-robin comparison on in-water bio-optical algorithms[J]. Remote Sensing of Environment, 2015, 162: 271-294.
Moore T S, Dowell M D, Bradt S, et al. An optical water type framework for selecting and blending retrievals from bio-optical algorithms in lakes and coastal waters[J]. Remote sensing of environment, 2014, 143: 97-111.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
library(FCMm) library(ggplot2) library(magrittr) library(stringr) data("Nechad2015") x <- Nechad2015[,3:11] wv <- gsub("X","",names(x)) %>% as.numeric set.seed(1234) w <- sample.int(nrow(x)) x <- x[w, ] names(x) <- wv nb = 4 # Obtained from the vignette "Cluster a new dataset by FCMm" set.seed(1234) FD <- FuzzifierDetermination(x, wv, do.stand=TRUE) result <- FCM.new(FD, nb, fast.mode = TRUE) p.spec <- plot_spec(result, show.stand=TRUE) print(p.spec$p.cluster.spec) Chla <- Nechad2015$X.Chl_a..ug.L.[w] Chla[Chla >= 999] <- NA dt_Chla <- run_all_Chla_algorithms(x) %>% as.data.frame dt_Chla <- data.frame(Chla_true = Chla, BR_Gil10 = dt_Chla$BR_Gil10, OC4_OLCI = dt_Chla$OC4_OLCI, OCI_Hu12 = dt_Chla$OCI_Hu12, NDCI_Mi12= dt_Chla$NDCI_Mi12) %>% round(3) w = which(!is.na(dt_Chla$Chla_true)) dt_Chla = dt_Chla[w,] memb = result$res.FCM$u[w,] %>% round(4) cluster = result$res.FCM$cluster[w] Asses_results <- Getting_Asses_results(sample.size=length(cluster), pred = dt_Chla[,-1], meas = data.frame(dt_Chla[,1]), memb = memb, cluster = cluster) Score = Scoring_system(Asses_results) # show the total score table knitr::kable(round(Score$Total_score, 2)) # Examples of `Scoring_system_bootstrap` set.seed(1234) Score_boo <- Scoring_system_bootstrap(Times = 3, Asses_results) # try to set large `Times` when using your own data # Show the bar plot of scores Score_boo$plot_col # Show the scatter plot of measure-estimation pairs Score_boo$plot_scatter # Show error metrics knitr::kable(round(Score_boo$metric_results$MAE, 2), caption = "MAE") knitr::kable(round(Score_boo$metric_results$CMAPE, 2), caption = "CAPE") knitr::kable(round(Score_boo$metric_results$BIAS, 2), caption = "BIAS") knitr::kable(round(Score_boo$metric_results$CMRPE, 2), caption = "CRPE") # you would see the blending estimations outperform than other candidates
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.