NEWS.md

In WuyangFF95/SynSigEval: Evaluate Results of Mutational Signature Analysis Software on Synthetic Spectra Created by Package SynSigGen

SynSigEval

Evaluate Results of Computational Approaches on Synthetic Datasets by generated by package SynSigGen.

NEWS

SynSigEval 0.4.0

Important changes on dependent packages

• Requires mSigTools >= 1.0.0
• No longer requires ICAMSxtra == 0.1.0, as the functions:
• ReadExposure() (for importing exposures)
• WriteExposure() (for exporting exposures)
• TP_FP_FN_avg_sim() have been moved to mSigTools >= 1.0.0
• No longer suggests mSigHdp

Changes in behavior

In function SummarizeSigOneSubdir(): Users can specify verbose >= 1 to receive diagnostic message when summarizing results of one run. The column "gt.sig" in the summary table is now renamed to "ref.sig". This summary table is available at: - File match.ex.to.gt.csv - Object sigAnalysis$table * The "ground-truth signature" has been changed to "reference signature" in all R files. Below are the changes of variable names: - sigAnalysis$gt.sigs => sigAnalysis$ref.sigs - sigMatch$gt.sigs => sigMatch$ref.sigs - gt.sigs => ref.sigs - ground.truth.sigs => reference.sigs Below are the changes of texts: - gt sig => reference signature - ground-truth signature => reference signature

Other than the above-mentioned changes, SynSigEval v0.4.0 has the same functionality as SynSigEval v0.3.1.

SynSigEval 0.3.1

Important changes

• Only the best one-to-one matches whose cosine similarities greater than threshold (0.9 by default) are counted into average cosine similarity,
• Use Hungarian algorithm to match extracted signatures to ground-truth signatures, in a one-to-one, bipartite fashion. This prevents multiple extracted signatures matching to one same ground-truth signature, and will ensure that two similar extracted signatures matching to two similar ground-truth signatures. Using Hungarian algorithm can correctly calculate the number of true positive, false negative, and false positive signatures.

SynSigEval 0.2.3

Fixed

• Fixed column name errors in SummarizeOneToolMultiDatasets().

Reshaped

• Reshaped code of SummarizeMultiToolsMultiDatasets() by:
  • Split codes into two internal functions SummarizeMultiToolsMultiDatasetsExtr() and SummarizeMultiToolsMultiDatasetsAttr()
  • Removed codes to plot violin plots. Bar charts are better representation compared to violin plots.

SynSigEval 0.2.2

Added

• Added violin plots and summary tables for:
  • Total number of signatures extracted
  • Number of extracted signatures with cosine similarity to each ground-truth signature > 0.9

Enabled

• Enabled sorting of computational approaches with mean of composite measure. (into ascending or descending order, by specifying sort.by.composite.extraction.measure = "ascending" or "descending")
• By specifying summarize.exp = FALSE, users can choose not to evaluate the performance of exposure inference, even when inferred.exposures.csv exists in a result folder.

Cleaned

• Moved SummarizeMultiToolsOneDataset.R into data-raw/deprecated.codes.
• Cleaned redundant files in "data-raw/", including SignatureAnalyzer.052418/ and scripts.for.SBS1SBS5/. Now these files are in package SynSigRun

SynSigEval 0.2.1

First version evaluates the results on SBS1-SBS5-only dataset series in a 4-layer folder structure:

Level 1: Original output (extracted signatures, inferred exposures, plots) by computational approaches (e.g. 3a.Original_output_K_unspecified);

Level 2: Files generated by one computational approach with multiple runs on each spectra datasets (e.g. TCSM.results);

Level 3: Results of computational approaches on each spectra dataset, may contain results of multiple runs with different random seeds (e.g. S.0.1.Rsq.0.1);

Level 4: Results of runs with specified computational approach, spectra dataset and random seed (e.g. seed.1, and run.1 for computational approaches whose random seeds cannot be specified by user).

Notes for 4-layer folder structure:

1. The synthetic tumor spectra (0.input_datasets) should be generated by CreateSBS1SBS5CorrelatedSyntheticData() in SynSigGen >= 1.0.6. Alternatively, you may download the compressed dataset through this link and decompress the zip folder.

2. We recommend running computational approaches with scripts in

   • <SynSigRun_HOME>/data-raw/scripts.for.SBS1SBS5/2a_running_approaches_K_unspecified
   • <SynSigRun_HOME>/data-raw/scripts.for.SBS1SBS5/2b_running_approaches_K_as_2 provided by SynSigRun >= 0.1.0. Before running, kindly set working directory to the parent directory of 0.input_datasets. The results will be placed into two folders:
   • <Working_Directory>/2a.Original_output_K_unspecified
   • <Working_Directory>/2b.Original_output_K_as_2

3. The results can be evaluated by scripts in

   • <SynSigRun_HOME>/data-raw/scripts.for.SBS1SBS5/3a_evaluation_K_unspecified
   • <SynSigRun_HOME>/data-raw/scripts.for.SBS1SBS5/3b_evaluation_K_as_2 provided by SynSigRun >= 0.1.0.The evaluation will be placed into two folders:
   • <Working_Directory>/1a.Top_level_summary_for_K_unspecified
   • <Working_Directory>/1b.Top_level_summary_for_K_as_2

4. Full results can be viewed at this link.

SynSigEval 0.1.9

Last version evaluates the results on SBS1-SBS5-only dataset in a 5-layer folder structure, using the functions listed below:

SummarizeSigOneExtrAttrSubdir
SummarizeMultiRuns
SummarizeOneToolMultiDatasets
SummarizeMultiToolsOneDataset
SummarizeMultiToolsMultiDatasets

This require the synthetic tumor datasets and the results of computational approaches in the 5-layer folder structure:

Level 1: Datasets (e.g. S.0.1.Rsq.0.1); Layer 2: Folder sp.sp:

• Meaning of 1st "sp": synthetic tumors were generated using signature profiles extracted by SigProfiler in Alexandrov et al., 2020. The signature profiles are available here.
• Meaning of 2nd "sp": according to Alexandrov et al., 2015, many tumor types have a mean SBS1 exposure of ~300, inferred by SigProfiler. Therefore, we chose the log10(exposure of SBS1) in 20 synthetic datasets to be 2.5.

Level 3: De-novo extraction (ExtrAttr), or extraction with ground-truth signatures known (ExtrAttrExact); Level 4: Results of computational approaches (e.g. hdp.results); Level 5: Results of runs with seeds (e.g. seed.1, run.1).

Notes for 5-layer folder structure:

• The synthetic tumor spectra should be generated by CreateSBS1SBS5CorrelatedSyntheticData() in SynSigGen = 1.0.4, and
• We recommend generate the results of computational approaches using data-raw/scripts.for.SBS1SBS5 in SynSigRun = 0.0.5.

SynSigEval 0.1.2

Updated package documentation and README.

SynSigEval 0.1.1

Removed

• MatchSigsAndRelabel.R (functionality was moved to ICAMSxtra) and adjusted the code in this package accordingly

SynSigEval 0.0.4.90xx

Added

• Created README.md and reference manual.

WuyangFF95/SynSigEval documentation built on Sept. 18, 2022, 11:41 a.m.