In irycisBioinfo/PATO: Pangenome Analysis Toolkit
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>"
)
{width=100%}
Installation
Linux/Unix
PATO uses external binaries so it can not be included in CRAN repository. To install PATO use \emph{devtools} package
install.packages("devtools")
Once you have installed \emph{devtools} install PATO from GitHub.
devtools::install_github("https://github.com/irycisBioinfo/PATO.git", build_vignettes = TRUE)
knitr::opts_chunk$set(
message = FALSE,
warning = FALSE,
include = FALSE
)
Introduction
PATO is a R package designed to analyze pangenomes (set of genomes) intra or inter species. It allows to analyze the core-genome, accessory genome and whole genome, the population structure, and the horizontal gene transfer dynamics. PATO uses, as core software, MASH, MMSeq2 and R.
These software can handle thousands of genomes using conventional computers without the necessity to use on a HPC facilities. PATO can analyze data in mash distance format (whole genome) or accnet format (accessory genome) Most of the functions can handle both objects. Some functions are specific for some kind of data. The primitive object mmseq is a orthologous clustering that is used to build accnet object, annotate the genomes and characterize the core-, accessory- and pan-genome size (and dynamic).
{width=100%}
Usage
Classes
PATO has been designed as a toolkit. Most of the functions use as input other outputs functions, creating a complete workflow to analysis large genome datasets. As R package, PATO can interact with other packages as ape, vegan, igraph or pheatmap. We have tried to developed PATO compatible with tidyverse packages in order to use %>% pipe command and to be compatible with ggplot2 and extensions (as ggtree).
In order to create a robust environment PATO uses specific data classes (S3 objects) to assure the compatibility among functions. PATO has 4 object classes:
mash objectmmseq objectaccnet objectnr_list object
Other outputs take external objects as igraph or ggplot.
The idea to have these object is to share a structured data among functions. These objects can be inspected and some of their data can be used with external function/packages.
mash class
A mash is a list of two element: table, matrix.
The first one, my_mash$table, is a square and symmetric matrix with the distances among genomes. As a matrix has genomes as rownames and colnames
The second one, my_mash$matrix is a data.table/data.frame with all the distances as list. The table has the columns c("Source","Target","Dist")
mmseq class
A mmseq object is a list of two elements: table and annot.
First element, my_mmseq$table, includes a data.table/data.frame with four columns c("Prot_genome", "Prot_Prot", "Genome_genome", "Genome_Prot"). This is the output of MMSeqs2 and described the clustering of the input genes/proteins. First column referees to the genome that contain the representative gene/protein of the cluster. Second one, is the representative protein of the cluster (i.e. the cluster name). Third column is the genome that includes the gene/protein of the fourth column.
In the second element, my_mmseq$annot, we can find a data.frame/data.table with the original annotation of all representative gene/protein of each cluster in two columns. The first one Prot_prot is the same that the second one of the first element.
accnet class
An accnet object is a list of three elements: list, matrix, annot.
First element, my_accnet$list, includes a data.frame/data.table with the columns c("Source","Target","degree") the correspondences among proteins/genes and genomes and the degree of the corresponding protein/gene. In some cases, for example, when the accnet object cames from accnet_with_padj function or pangenomes the third column can be different and includes information such us the frequency of the protein/gene in the pangenome (in the case of pangenomes), or the p-value of the association between the genome and the protein/gene (in the case of accnet_with_padj)
Second element, my_accnet$matrix, is a presence/absence matrix. It is very important to know that it has a column Source, so in the case of use the matrix out of PATO you should convert this column in rownames (my_matrix <- my_accnet$matrix %>% column_to_rownames("Source"))
nr_list class
A nr_list is a S3 object of three elements: Source,centrality and cluster.
nr_list can be coerced to a data.frame of three columns using as.data.frame function. Element Source has the accession/name of the genomes, centrality has the centrality value of the genome, according to the degrees of vertices, for its cluster. Finally, cluster has the membership of the Source genome.
Main Functions
PATO has two main data classes, mash and mmseq. MASH is a whole-genome comparison tool that can compare
thousands of genomes in a few minutes (even seconds).
On other hand, MMSeq2 is a ultra fast software for sensitive search and clustering genes or proteins. Since last versions of MMSeq2 it include lin-clust method that allow to cluster huge data sets in linear time (most of the method are quadratic time or log(n)²) but with high identities.
Both MASH and MMSeq2 allow multi-threading to accelerate the processing.
MASH
Mash is a method for "Fast genome and metagenome distance estimation using MinHash"(). MASH accept nucleotide or aminoacid fasta files and estimated a similarity distance. To extend the information we recommend to read the main paper Mash: fast genome and metagenome distance estimation using MinHash. Ondov BD, Treangen TJ, Melsted P, Mallonee AB, Bergman NH, Koren S, Phillippy AM. Genome Biol. 2016 Jun 20;17(1):132. doi: 10.1186/s13059-016-0997-x
To create a mash object you need to supplie a list (a vector indeed) of files. By default mash use Amino Acids fasta files.
my_mash <- mash(my_list, type ="prot")
Besides, you can change some parameter as n_cores, sketch, kmer and type.
my_mash <- mash(my_list, n_cores = 20, sketch = 1000, kmer = 21, type = "prot")
MMSeqs2
Creators of MMSeqs define their software as:
MMseqs2 (Many-against-Many sequence searching) is a software suite to search and cluster huge protein and nucleotide sequence sets. MMseqs2 is open source GPL-licensed software implemented in C++ for Linux, MacOS, and (as beta version, via cygwin) Windows. The software is designed to run on multiple cores and servers and exhibits very good scalability. MMseqs2 can run 10000 times faster than BLAST. At 100 times its speed it achieves almost the same sensitivity. It can perform profile searches with the same sensitivity as PSI-BLAST at over 400 times its speed.
To create a mmseq object you just have to run:
my_mmseq <- mmseqs(my_list)
You can change default parameter as coverage, identity or evalue among n_cores. Also you can set especifict parameters of clustering (searching for protein families and orthologous). These parameters are cov_mode and cluster_mode.
my_mmseq <- mmseqs(my_list, coverage = 0.8, identity = 0.8, evalue = 1e-6, n_cores = 20, cov_mode = 0, cluster_mode = 0)
The mmseq object is key in the analysis of the pangenome definition (core, accessory, pangenome) as well as other process such us annotation or population structure (if you are using accessory to defined it).
Coverage Mode
(this section has been extracted from https://github.com/soedinglab/MMseqs2/wiki)
MMseqs2 has three modes to control the sequence length overlap "coverage": (0) bidirectional, (1) target coverage, (2) query coverage and (3) target-in-query length coverage. In the context of cluster or linclust, the query is seen representative sequence and target is a member sequence. The -cov-mode flag also automatically sets the cluster_mode.
{width=100%}
Clustering modes
All clustering modes transform the alignment results into an undirected graph. In this graph notation, each vertex (i.e. node) represents a sequence, which is connected to other sequences by edges. An edge between a pair of sequences is introduced if the alignment criteria (e.g. identity, coverage and evalur) are fulfilled.
The Greedy Set cover (cluster-mode = 0) algorithm is an approximation for the NP-complete optimization problem called set cover.
{width=70%}
Greedy set cover works by interactively selecting the node with most connections and all its connected nodes to form a cluster and repeating until all nodes are in a cluster.
The greedy set cover is followed by a reassignment step. A Cluster member is assigned to another cluster centroid if their alignment score was higher.
Connected component (cluster_mode = 1) uses transitive connection to cover more remote homologous.
{width=70%}
In connected component clustering starting at the mostly connected vertex, all vertices that are reachable in a breadth-first search are members of the cluster.
Greedy incremental (cluster_mode = 2) works analogous to CD-HIT clustering algorithm.
{width=70%}
Greedy incremental clustering takes the longest sequence (indicated by the size of the node) and puts all connected sequences in that cluster, then repeatedly the longest sequence of the remaining set forms the next cluster.
AccNET
Accnet is a representation of the "accessory genome" using a bipartite network. (see more in Lanza, V. F., Baquero, F., de la Cruz, F. & Coque, T. M. AcCNET (Accessory Genome Constellation Network): comparative genomics software for accessory genome analysis using bipartite networks. Bioinformatics 33, 283–285 (2017)). Accnet creates a graph using two kind of nodes: genomes and proteins/gene. One genome is connected to one proteins if the genome has this protein (i.e. a protein that belong to this protein family/cluster). The accnet object is other of the main data type in PATO. PATO can create some representations of the accessory genome (dendrograms, networks, plots....) and has some functions to analyze the accessory genome properly. The function accnet() builds an accnet object taking into account the maximun frequency of each protein/gene to be considered as part of accessory genome. Moreover, the user can decide to include single protein/genes or not (those which are only present in one sample).
Accnet depends of the definition of protein families so the input of the accnet() function is a mmseqs object. The definition of the homologous cluster is critical to define the accessory genome.
accnet() can be called as:
my_accnet <- accnet(my_mmseq, threshold = 0.8, singles = TRUE)
or piping the mmseqs() function
my_accnet <- mmseqs(my_files) %>% accnet(threshold = 0.8)
Workflow
This is an example analyzing 2.941 high quality Escherichia coli and Shigella downloaded from NCBI Assembly database (you can download a copy from https://doi.org/10.6084/m9.figshare.13049654). In this case we'll work with protein multi-fasta files (i.e. *.faa).
PATO can handle both nucleotide or aminoacid multi-fasta files. PATO does not use contigs fasta file. We recomend to use PROKKA output files .faa or .ffn. If you are using NCBI, EBI or other public database note that there are especific files for multi-fasta genes file or multi-fasta protein files. It is important to know that PATO take as annotation the headers of the fasta sequence, and usually, protein annotation is better than gen (CDS) annotation. Thus, we recommend to use protein fasta files.
It is very important to establish the working directory. We recommend to set the working directory in the folder where the multi-fasta files are. Make sure that you have enough free disk space.
library(pato)
# We strongly recommend to load tidyverse meta-package
library(tidyverse)
setwd("/myFolder/")
The main input file for PATO is a plain-text list of files with the path to the multi-fasta files. This list
could be easily made in your Unix console.
```{bash eval=FALSE, include = TRUE}
ls *.faa > my_list.txt
... and import to R enviroment.
```r
files <- read.table("my_list.txt",header = FALSE)
Quality Control
PATO has implemented a classification function to identify the most probable species
determination for each input file. The method is based in the minimun mash distance of each
file against a homemade reference database. Thr reference database comprises all reference and representative genomes of Refseq. Input files can be fasta nucleotides or aminoacids.
species <- classifier(files,n_cores = 20,type = 'prot')
species %>% group_by(organism_name) %>% summarise(Number = n())
# A tibble: 8 x 2
organism_name Number
<chr> <int>
1 Citrobacter gillenii 1
2 Escherichia coli O157:H7 str. Sakai 323
3 Escherichia coli str. K-12 substr. MG1655 2520
4 Escherichia marmotae 4
5 Shigella boydii 18
6 Shigella dysenteriae 7
7 Shigella flexneri 2a str. 301 45
8 Shigella sonnei 40
In this example we found 5 foreign genomes (1 Citrobacter and 4 E. marmotae). Now we can create a new list of files with the output of classifier. Using the tidyverse style we filter the unwanted sequences
files = species %>% filter(!grepl("Citrobacter",organism_name)) %>% filter(!grepl("marmotae",organism_name))
Now we create the main objects mashand mmseqs. We go to create a protein families with 80% of identity, 80% of coverage and a maximun of 1e-6 of Evalue.
ecoli_mash_all <- mash(files, n_cores = 20, type = 'prot')
ecoli_mm<- mmseqs(files, coverage = 0.8, identity = 0.8, evalue = 1e-6, n_cores = 20)
In my case, I have 20 cores in my computer.
Finally we can create an accnet object:
ecoli_accnet_all <- accnet(ecoli_mm,threshold = 0.8, singles = FALSE)
Pre-processing
Outliers
One of the things that happen when you download genomes from public databases is that some times you can find some outliers, genomes that not belong to the selected specie. PATO implement a function to indentify outliers and to remove it if is neccessary. In this case we set a threshold of 0.06 (~94% of ANI) to be considered as outlier.
outl <- outliers(ecoli_mash_all,threshold = 0.06)
You can remove the outliers from mash or accnet objects
ecoli_mash_all <-remove_outliers(ecoli_mash_all, outl)
ecoli_accnet_all <-remove_outliers(ecoli_accnet_all, outl)
(see )
If you desire to remove outliers from files object you can do it
files <- anti_join(files, outl, by=c("V1"="Source"))
This command remove the outliers from the original files object. We recommend do it before mmseqs command.
Redundancies
Most of the times in the public databases are redundancies. The inclusion of outbreaks and studies about specific Sequence Types or Sub-clones in the databases bias the real diversity of the species. PATO implement a function to select a non-redundant subset based on a fix distance (similarity), a % of samples or a selected number of genomes. It is an iterative approach so we implement a threshold of toerance due to some time find the exact solution is impossible. This function accepts mash or accnet object as input. Among, there is a function to create mash or accnet objects extracting the subset of genomes
# For select 800 non redundant samples
nr_list <- non_redundant(ecoli_mash_all,number = 800)
or
# For select a subset of samples with 99.99% of indentity
nr_list <- non_redundant(ecoli_mash_all, distance = 0.0001)
to create the objects only with the representatives of each cluster:
ecoli_accnet <- extract_non_redundant(ecoli_accnet_all, nr_list)
ecoli_mash <- extract_non_redundant(ecoli_mash_all, nr_list)
Moreover, there is another function non_redundant_hier() that perform a hierarchical approach to perform the clusterization of the genomes. It is faster for big dataset (>1000 genomes) so we recommend use it in this cases.
nr_list <- non_redundant_hier(ecoli_mash_all,800, partitions = 10)
ecoli_accnet <- extract_non_redundant(ecoli_accnet_all, nr_list)
ecoli_mash <- extract_non_redundant(ecoli_mash_all, nr_list)
The partition parameter set the number of hierarchical steps.
Core Genome analysis
PATO implements a set of tools to inspect the core genome. It includes the pangenome composition (core, accessory, pangenome) and the function to create a core-genome alignment.
We can inspect the Pangenome composition of our dataset with:
cp <- core_plots(ecoli_mm,reps = 10, threshold = 0.95, steps = 10)
PATO can build a core-genome alignment to use with external software such as IQTree[http://www.iqtree.org/], RAxML-NG[http://www.exelixis-lab.org/software.html], FastTree[http://www.microbesonline.org/fasttree/] or other phylogenetic inference software. The core genome is computed based on a mmseq object, so the definition of core-genome depends of the parameters used in that step.
In this example we build a core-genome aligment of the ~800 samples of non-redundant result.
nr_files = nr_list %>%
as.data.frame() %>%
group_by(cluster) %>%
top_n(1,centrality) %>%
summarise_all(first) %>%
select(Source) %>%
distinct()
We have selected the best sample of each cluster (max centrality value). Some cluster has several samples with the same centrality value, so we take one of them (the first one). Now we run the core genome for this list of samples.
core <- mmseqs(nr_files) %>% core_genome(type = "prot")
export_core_to_fasta(core,"core.aln")
Some time, when you are using public data, core-genome can be smaller than you expect. Citating Andrew Page, creator of Roary (https://sanger-pathogens.github.io/Roary/)
I downloaded a load of random assemblies from GenBank. Why am I seeing crazy results?
Gene prediction software rarely completely agrees, with differing opinions on the actual start of a gene, or which of the overlapping open reading frames is actually the correct one, etc. As a result, if you mix these together you can get crazy results. The solution is to reannotate all of the genomes with a single method (like PROKKA). Otherwise you will waste vast amounts of time looking at noise, errors, and the batch effect.
other times you can have a problem of outliers
The exported files are in Multi-alignment FASTA format and ca be use with most of the Phylogenetic tools. In this case we used FastTree for phylogenetic inference
```{bash eval=FALSE, include=TRUE}
fasttreeMP core.aln > core.tree
And then we can read the output file to import to our R environment and plot the result.
```r
library(phangorn)
library(ggtree)
core_tree = read.tree("core.tree")
annot_tree = species %>%
filter(!grepl("Citrobacter",organism_name)) %>%
filter(!grepl("marmotae",organism_name)) %>%
select(Target,organism_name) %>% distinct()
core_tree %>% midpoint %>% ggtree(layout = "circular") %<+% annot_tree + geom_tippoint(aes(color = organism_name))
In this case we have added the species information extracted above.
You must take into account that Maximun Likelyhood tree can take long computational times.
Finally, you can obtain a SNPs matrix from the coregenome alignment. The matrix shown the number of total SNPs (or changes in the case of proteins alignments) shared among the samples. The matrix can be normalized by the total length of the genome (in magabases) or can be raw numer of variants.
var_matrix = core_snps_matrix(core, norm = TRUE)
pheatmap::pheatmap(var_matrix,show_rownames = F, show_colnames = F)
Accessory Genome Analysis
I this case we are creating the accessory genome taking those proteins presence in no more than 80% of the genomes.
ecoli_accnet_all <- accnet(ecoli_mm,threshold = 0.8, singles = FALSE)
As we have shown above, PATO has an object type to accessory genome accnet and functions to analyze and visualize the content of the accessory genome and the relationship between genes/proteins and the genomes. We can visualize an accnet object as a bipartite network. Commonly, AccNET networks are very large, so, we recommend to visualize the networks using Gephi. We do not recommend to try visualize AccNET networks with more than 1.000 genomes.
One of the most interesting things when we analyze an accessory genome is try to find what genes/proteins are over-represented in a set of genomes. accnet_enrichment_analysis() function analyze what genes/proteins are over-represented in a cluster in comparisson with the population (i.e. in the wholem dataset). The clusters definition can be external (any catagorical metadata such as Sequence Type, Source, Serotype etc...) or internal using some clustering proccess. We have to take into account, that the redundancies can bias this kind of analysis. If there are an over-representation of samples, for example an outbreak, in your dataset the results could be bias. You will find more significant genes/proteins because the diversity of the dataset is not homegenously distributed. For this reason we recommend to use a non redundant set of samples. In this example we select 800 non redundant genomes.
ec_nr<- non_redundant(ecoli_mash,number = 800 )
ecoli_accnet_nr <- extract_non_redundant(ecoli_accnet, ef_nr)
ec_800_cl <- clustering(ecoli_accnet_nr, method = "mclust", d_reduction = TRUE)
Now we can visualize the network using Gephi.
export_to_gephi(ecoli_accnet_nr, "accnet800", cluster = ec_800_cl)
[https://doi.org/10.6084/m9.figshare.13089338]
To perform the enrichment analysis we use:
accnet_enr_result <- accnet_enrichment_analysis(ecoli_accnet_nr, cluster = ef_800_cl)
# A tibble: 1,149,041 x 14
Target Source Cluster perClusterFreq ClusterFreq ClusterGenomeSize perTotalFreq TotalFreq OdsRatio pvalue padj AccnetGenomeSize AccnetProteinSi… Annot
<chr> <chr> <dbl> <dbl> <int> <int> <dbl> <int> <dbl> <dbl> <dbl> <int> <int> <chr>
1 1016|NZ_CP053592.… GCF_000009565.1_ASM956v… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
2 1016|NZ_CP053592.… GCF_000022665.1_ASM2266… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
3 1016|NZ_CP053592.… GCF_000023665.1_ASM2366… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
4 1016|NZ_CP053592.… GCF_000830035.1_ASM8300… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
5 1016|NZ_CP053592.… GCF_000833145.1_ASM8331… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
6 1016|NZ_CP053592.… GCF_001039415.1_ASM1039… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
7 1016|NZ_CP053592.… GCF_001596115.1_ASM1596… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
8 1016|NZ_CP053592.… GCF_009663855.1_ASM9663… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
9 1016|NZ_CP053592.… GCF_009832985.1_ASM9832… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
10 1016|NZ_CP053592.… GCF_013166955.1_ASM1316… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 ""
# … with 1,149,031 more rows
[https://doi.org/10.6084/m9.figshare.13089314.v2]
Now, we can export a new network with the adjusted p-values as edge-weigth.
accnet_with_padj(accnet_enr_result) %>% export_to_gephi("accnet800.padj", cluster = ec_800_cl)
[https://doi.org/10.6084/m9.figshare.13089401.v1]
PATO also include some functions to study the genes/proteins distribution: singles(), twins() and conicidents().
singles() finds those genes/proteins that are only present in a sample (genome, pangenome...).twins() finds those genes/proteins that have the same connections (i.e. genes/proteins present in the same genomes).coincidents() finds those genes/proteins with similar connections (i.e. genes/proteins that usually are together)
The differencies between twins() and concidents() are that coincidents() is more flexible and therefore less sensitive to outliers. twins() is faster and accurate but is very sensitive to noise or outliers because just one missing conection (for example a bad prediction of a protein in a genome) remove automatically that proteins from the twin group. On other hand, coincidents() is slower, and some time the results are to much generalistic giving big cluster of pseudo-core.
Population structure
PATO includes tools for population structure search and comparisson. PATO can analyse the population structure of the whole-genome (MASH based) or the accessory structure (AccNET based)
We can visualize our dataset as a dendrogram (i.e a tree). PATO allows to visualize both mash and accnet data as a tree. In the caso of accnet data, first PATO calculate a distance matrix using the presence/absence matrix in combination with Jaccard distance. The function similarity_tree() implement different methods to build the dendrogram. Phylogenetic aproaches such as Neighbour Joining, FastME:Minimun Evolution and hierarchical clustering method such us complete linkage, UPGMA, Ward's mininum variance, and WPGMC.
mash_tree_fastme <- similarity_tree(ecoli_mash)
mash_tree_NJ <- similarity_tree(ecoli_mash, method = "NJ")
mash_tree_upgma <- similarity_tree(ecoli_mash,method = "UPGMA")
accnet_tree_upgma <- similarity_tree(ecoli_accnet,method = "UPGMA")
The output has a phylo format, so can be visualize with external packages as ggtree.
mash_tree_fastme %>% midpoint %>% ggtree(layout = "circular") %<+% annot_tree + geom_tippoint(aes(color = organism_name))
Using others external packages, we can compare the arragement of the pangenom (mash data) against the accessory genome (accnet)
library(dendextend)
tanglegram(ladderize(mash_tree_upgma), ladderize(accnet_tree_upgma), fast = TRUE, main = "Mash vs AccNET UPGMA")
Some Maximun Likelyhood inference trees software accept,as input, binary data (0-1) alignments.So, we can use accessory data (accnet) to infere a tree (non-phylogenetic) with this data. This is similar to the similarity_tree() but instead to be based on distance metrics it's based on ML principles. To export this alignment you can use export_accnet_aln().
mmseqs(files) %>% accnet() %>% export_accnet_aln(.,file ="my_accnet_aln.fasta")
And then you can use it as input alignment
```{bash eval=FALSE, include=FALSE}
iqtree -s my_accnet_aln.fasta -st BIN
Again, we can import to R and plot it.
```r
acc_tree = read.tree("acc.aln.treefile")
acc_tree %>% midpoint.root() %>% ggtree()
This kind of alignments can be very large. Most of the times accessory genomes contains a lot of spurious genes/proteins that do not add information to the alignment. For this reason export_accnet_aln() has a parameter, min_freq, to filter the genes/proteins by their frequency. This option cut significantly the alingment length and improves computational times.
PATO has a set of function to visualize data. We have saw the tree but algo implements methods to visualize as networks the relationships among genomes. K-Nearest Neighbour Networks is a representation of the realtionship of the data in a network plot. This function (knnn()) builds a network linking each genome with their K best neighbours instead of chose a distance threshold. This approach minimize the clutering of the network and tries to create a connected network. The method can chose the K best neighbours with or without repetitions. That means that if repeats=FALSE each node are linked with the best K neighbours that does be not linked yet with the node.
# K-NNN with 10 neighbours and with repetitions
knnn_mash_10_w_r <- knnn(ecoli_mash,n=10, repeats = TRUE)
# K-NNN with 25 neighbours and with repetitions
knnn_mash_25_w_r <- knnn(ecoli_mash,n=25, repeats = TRUE)
# K-NNN with 50 neighbours and with repetitions
knnn_mash_50_w_r <- knnn(ecoli_mash,n=50, repeats = TRUE)
knnn() function returns an igraph object that can be visualize with several packages. However, PATO implements its own function plot_knnn_network() that uses threejs and the own igraph for layouting and plotting the network. Due to the common size of the network we strongly recommend to use this function or use the external software Gephi (https://gephi.org/). You can use the function export_to_gephi() to export these networks to Gephi.
export_to_gephi(knnn_mash_50_w_r,file = "knnn_50_w_r.tsv")
If you want, you can use the internal function plot_knnn_network() to visualize the network. This funtion uses igraph layouts algorith to arrange the network and threejs package to draw and explore the network.
plot_knnn_network(knnn_mash_50_w_r)
PATO includes a different clustering methods for different kind of data (objects). clustering() function joins all the different approaches to make easier the process. For mash and accnetobjects we have the following methods:
- mclust: It perform clustering using Gaussian Finite Mixture Models. It could be combine with d_reduction. This method uses Mclust package. It has been implemented to find the optimal cluster number
- upgma: It perform a Hierarchical Clustering using UPGMA algorithm. The user must provide the number of cluster.
- ward.D2: It perform a Hierarchical Clustering using Ward algorithm. The user must provide the number of cluster.
- hdbscan: It perform a Density-based spatial clustering of applications with noise using DBSCAN package. It find the optimal number of cluster.
Any of the above methods is compatible with a multidimentsional scaling (MDS). PATO performs the MDS using UMAP algorithm (Uniform Manifold Approximation and Projection). UMAP is a tools for MDS, similar to other machine learning MDS technics as t-SNE or Isomap. Dimension reduction algorithms tend to fall into two categories those that seek to preserve the distance structure within the data and those that favor the preservation of local distances over global distance. Algorithms such
as PCA , MDS , and Sammon mapping fall into the former category while t-SNE, Isomap or UMAP,fall into the later category. This kind of MDS combine better with the clustering algorithms due to clustering process try to find local structures.
ec_cl_mclust_umap <- clustering(ecoli_mash, method = "mclust",d_reduction = TRUE)
On other hand, clustering() can handle knnn networks as input. In this caso, PATO uses network clustering algorithms such us:
- greedy: Community structure via greedy optimization of modularity
- louvain: This method implements the multi-level modularity optimization algorithm for finding community structure
- walktrap: Community strucure via short random walks
ec_cl_knnn <-clustering(knnn_mash_50_w_r, method = "louvain")
Whatever the method used, the output has allways the same structure: a data.frame with two columns c("Source","Cluster"). The reason of this format is to be compatible with the rest of the data, being able to combine with the rest of the objects using Source variable as key.
To visualize the clustering data or just to see the data structure we can use umap_plot() function. The functions performs a umap reduction and plot the results. We can include the clustering results as a parameter and visualize it.
umap_mash <- umap_plot(ecoli_mash)
umap_mash <- umap_plot(ecoli_mash, cluster = ef_cl_mclust_umap)
We also can use plot_knnn_network() to visualize the network clustering results.
cl_louvain = clustering(knnn_mash_25_wo_r, method = "louvain")
plot_knnn_network(knnn_mash_25_wo_r, cluster = cl_louvain, edge.alpha = 0.1)
Annotation
PATO has a function to annotate the Antibiotic Resistance Genes and the Virulence Factor: annotate(). Antibiotic resistance are predicted using MMSeqs2 over ResFinder database (https://cge.cbs.dtu.dk/services/ResFinder/) (doi: 10.1093/jac/dks261). For Virulence Factors we use VFDB (http://www.mgc.ac.cn/cgi-bin/VFs/v5/main.cgi)(doi: 10.1093/nar/gky1080). VFBD has two sets of genes VF_A the core-dataset and VF_B the full dataset. Core dataset includes genes associated with experimentally verified VFs only, whereas full dataset covers all genes related to known and predicted VFs in the database.
annotate() results is a table with all positive hits of each gene/protein of the dataset (files). annotate() can re-use the results of mmseqs() to accelerate the proccess. In the same way, the query can be all genes/proteins or accessory genes/proteins. The results are quite raw so user must curate the table. We recommend to use the tidyverse tools. It is very easy to obtain a greatfull results.
library(tidyverse)
annotation <- annotate(files, type = "prot",database = c("AbR","VF_A"))
annotation %>%
filter(pident > 0.95 ) %>% #remove all hits with identity lower than 95%
filter(evalue < 1e-6) %>% #remove all hits with E-Value greater than 1e-6
group_by(Genome,Protein) %>% arrange(desc(bits)) %>% slice_head() #select only the best hit for each protein-genome.
PATO includes also a function to create a heapmap with the annotation:
heatmap_of_annotation(annotation %>% filter(DataBase =="AbR"), #We select only "AbR" results
min_identity = 0.99)
Or to visualizae as a network.
network_of_annotation(annotation %>% filter(DataBase =="AbR"), min_identity = 0.99) %>% export_to_gephi("annotation_Network")
[https://doi.org/10.6084/m9.figshare.13121795]
Pangenomes Analysis
PATO is designed even to analyze the relationship among pangenomes of different (or not) species. Thats mean to analyze cluster of genomes (pangenomes) among then as individual elements. In this example we analyze 49.591 firmicutes genomes from NCBI data base. For that kind of large datasets it is recommended to use the specific pipeline for pangenomes. This pipeline first cluster the genomes into pangenomes, for example, cluster of species or intra-specie phylogroups or even sequence types (ST). The diversity of each cluster depends on the parameter distance. The pipeline creates homogeneous (in phylogenetic distance) clusters. Then the pipeline produces a accnet object so all the above described function can be use with this object as a common accnet object. The pipeline also include parameters to set the minimum amount of genomes to consider a pangenome and the minimum frequency of a protein/gene family to be included in a pangenome.
res <- pangenomes_from_files(files,distance = 0.03,min_pange_size = 10,min_prot_freq = 2)
export_to_gephi(res,"/storage/tryPATO/firmicutes/pangenomes_gephi")
In this case we produce a gephi table to visulize the accnet network of the dataset. To annotate the network we use the NCBI assembly table and takes the species name of each pangenome cluster.
assembly = data.table::fread("ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/assembly_summary.txt",sep = "\t",skip = 1, quote = "")
colnames(assembly) = gsub("# ","",colnames(assembly))
annot = res$members %>%
mutate(file = basename(path)) %>%
separate(file,c("GCF","acc","acc2"),sep = "_", remove = FALSE) %>%
unite(assembly_accession,GCF,acc,sep = "_") %>%
left_join(assembly) %>%
separate(organism_name,c("genus","specie"), sep = " ") %>%
group_by(pangenome,genus,specie) %>%
summarise(N = n()) %>%
distinct() %>%
group_by(pangenome) %>%
top_n(1,N) %>%
mutate(ID = paste("pangenome_",pangenome,"_rep_seq.fasta", sep = "",collapse = "")) %>%
write_delim("/storage/tryPATO/firmicutes/pangenomes_gephi_extra_annot.tsv",delim = "\t", col_names = TRUE)
We chose as species name the specie most frequent in each cluster. Then usin Gephi we visualiza the network.
{width=100%}
Other way to visualize the results is a heatmap of the sharedness among the pangenoms. PATO includes the function sharedness to visualize this result. To visualize the data.table result from sharedness() function we can use the package pheatmap. As the input value of pheatmap can be only a matrix we must first transform our result into a matrix:
sh <- sharedness(res)
sh <- sh %>% as.data.frame() %>%
rownames_to_column("ID") %>%
inner_join(annot) %>%
unite(Name,genus,specie,pangenome, sep = "_") %>%
select(-ID,-N)%>%
column_to_rownames("Name")
colnames(sh) = rownames(sh)
Now we can execute pheatmap() with our pangenome dataset.
pheatmap::pheatmap(sh,clustering_method = "ward.D2", clustering_distance_cols = "correlation", clustering_distance_rows = "correlation")
irycisBioinfo/PATO documentation built on Oct. 19, 2023, 3:07 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
{width=100%}
Installation
Linux/Unix
PATO uses external binaries so it can not be included in CRAN repository. To install PATO use \emph{devtools} package
install.packages("devtools")
Once you have installed \emph{devtools} install PATO from GitHub.
devtools::install_github("https://github.com/irycisBioinfo/PATO.git", build_vignettes = TRUE)
knitr::opts_chunk$set( message = FALSE, warning = FALSE, include = FALSE )
Introduction
PATO is a R package designed to analyze pangenomes (set of genomes) intra or inter species. It allows to analyze the core-genome, accessory genome and whole genome, the population structure, and the horizontal gene transfer dynamics. PATO uses, as core software, MASH, MMSeq2 and R.
These software can handle thousands of genomes using conventional computers without the necessity to use on a HPC facilities. PATO can analyze data in mash distance format (whole genome) or accnet format (accessory genome) Most of the functions can handle both objects. Some functions are specific for some kind of data. The primitive object mmseq is a orthologous clustering that is used to build accnet object, annotate the genomes and characterize the core-, accessory- and pan-genome size (and dynamic).
{width=100%}
Usage
Classes
PATO has been designed as a toolkit. Most of the functions use as input other outputs functions, creating a complete workflow to analysis large genome datasets. As R package, PATO can interact with other packages as ape, vegan, igraph or pheatmap. We have tried to developed PATO compatible with tidyverse packages in order to use %>% pipe command and to be compatible with ggplot2 and extensions (as ggtree).
In order to create a robust environment PATO uses specific data classes (S3 objects) to assure the compatibility among functions. PATO has 4 object classes:
mashobjectmmseqobjectaccnetobjectnr_listobject
Other outputs take external objects as igraph or ggplot.
The idea to have these object is to share a structured data among functions. These objects can be inspected and some of their data can be used with external function/packages.
mash class
A mash is a list of two element: table, matrix.
The first one, my_mash$table, is a square and symmetric matrix with the distances among genomes. As a matrix has genomes as rownames and colnames
The second one, my_mash$matrix is a data.table/data.frame with all the distances as list. The table has the columns c("Source","Target","Dist")
mmseq class
A mmseq object is a list of two elements: table and annot.
First element, my_mmseq$table, includes a data.table/data.frame with four columns c("Prot_genome", "Prot_Prot", "Genome_genome", "Genome_Prot"). This is the output of MMSeqs2 and described the clustering of the input genes/proteins. First column referees to the genome that contain the representative gene/protein of the cluster. Second one, is the representative protein of the cluster (i.e. the cluster name). Third column is the genome that includes the gene/protein of the fourth column.
In the second element, my_mmseq$annot, we can find a data.frame/data.table with the original annotation of all representative gene/protein of each cluster in two columns. The first one Prot_prot is the same that the second one of the first element.
accnet class
An accnet object is a list of three elements: list, matrix, annot.
First element, my_accnet$list, includes a data.frame/data.table with the columns c("Source","Target","degree") the correspondences among proteins/genes and genomes and the degree of the corresponding protein/gene. In some cases, for example, when the accnet object cames from accnet_with_padj function or pangenomes the third column can be different and includes information such us the frequency of the protein/gene in the pangenome (in the case of pangenomes), or the p-value of the association between the genome and the protein/gene (in the case of accnet_with_padj)
Second element, my_accnet$matrix, is a presence/absence matrix. It is very important to know that it has a column Source, so in the case of use the matrix out of PATO you should convert this column in rownames (my_matrix <- my_accnet$matrix %>% column_to_rownames("Source"))
nr_list class
A nr_list is a S3 object of three elements: Source,centrality and cluster.
nr_list can be coerced to a data.frame of three columns using as.data.frame function. Element Source has the accession/name of the genomes, centrality has the centrality value of the genome, according to the degrees of vertices, for its cluster. Finally, cluster has the membership of the Source genome.
Main Functions
PATO has two main data classes, mash and mmseq. MASH is a whole-genome comparison tool that can compare
thousands of genomes in a few minutes (even seconds).
On other hand, MMSeq2 is a ultra fast software for sensitive search and clustering genes or proteins. Since last versions of MMSeq2 it include lin-clust method that allow to cluster huge data sets in linear time (most of the method are quadratic time or log(n)²) but with high identities.
Both MASH and MMSeq2 allow multi-threading to accelerate the processing.
MASH
Mash is a method for "Fast genome and metagenome distance estimation using MinHash"(). MASH accept nucleotide or aminoacid fasta files and estimated a similarity distance. To extend the information we recommend to read the main paper Mash: fast genome and metagenome distance estimation using MinHash. Ondov BD, Treangen TJ, Melsted P, Mallonee AB, Bergman NH, Koren S, Phillippy AM. Genome Biol. 2016 Jun 20;17(1):132. doi: 10.1186/s13059-016-0997-x
To create a mash object you need to supplie a list (a vector indeed) of files. By default mash use Amino Acids fasta files.
my_mash <- mash(my_list, type ="prot")
Besides, you can change some parameter as n_cores, sketch, kmer and type.
my_mash <- mash(my_list, n_cores = 20, sketch = 1000, kmer = 21, type = "prot")
MMSeqs2
Creators of MMSeqs define their software as:
MMseqs2 (Many-against-Many sequence searching) is a software suite to search and cluster huge protein and nucleotide sequence sets. MMseqs2 is open source GPL-licensed software implemented in C++ for Linux, MacOS, and (as beta version, via cygwin) Windows. The software is designed to run on multiple cores and servers and exhibits very good scalability. MMseqs2 can run 10000 times faster than BLAST. At 100 times its speed it achieves almost the same sensitivity. It can perform profile searches with the same sensitivity as PSI-BLAST at over 400 times its speed.
To create a mmseq object you just have to run:
my_mmseq <- mmseqs(my_list)
You can change default parameter as coverage, identity or evalue among n_cores. Also you can set especifict parameters of clustering (searching for protein families and orthologous). These parameters are cov_mode and cluster_mode.
my_mmseq <- mmseqs(my_list, coverage = 0.8, identity = 0.8, evalue = 1e-6, n_cores = 20, cov_mode = 0, cluster_mode = 0)
The mmseq object is key in the analysis of the pangenome definition (core, accessory, pangenome) as well as other process such us annotation or population structure (if you are using accessory to defined it).
Coverage Mode
(this section has been extracted from https://github.com/soedinglab/MMseqs2/wiki)
MMseqs2 has three modes to control the sequence length overlap "coverage": (0) bidirectional, (1) target coverage, (2) query coverage and (3) target-in-query length coverage. In the context of cluster or linclust, the query is seen representative sequence and target is a member sequence. The -cov-mode flag also automatically sets the cluster_mode.
{width=100%}
Clustering modes
All clustering modes transform the alignment results into an undirected graph. In this graph notation, each vertex (i.e. node) represents a sequence, which is connected to other sequences by edges. An edge between a pair of sequences is introduced if the alignment criteria (e.g. identity, coverage and evalur) are fulfilled.
The Greedy Set cover (cluster-mode = 0) algorithm is an approximation for the NP-complete optimization problem called set cover.
{width=70%}
Greedy set cover works by interactively selecting the node with most connections and all its connected nodes to form a cluster and repeating until all nodes are in a cluster. The greedy set cover is followed by a reassignment step. A Cluster member is assigned to another cluster centroid if their alignment score was higher.
Connected component (cluster_mode = 1) uses transitive connection to cover more remote homologous.
{width=70%}
In connected component clustering starting at the mostly connected vertex, all vertices that are reachable in a breadth-first search are members of the cluster.
Greedy incremental (cluster_mode = 2) works analogous to CD-HIT clustering algorithm.
{width=70%}
Greedy incremental clustering takes the longest sequence (indicated by the size of the node) and puts all connected sequences in that cluster, then repeatedly the longest sequence of the remaining set forms the next cluster.
AccNET
Accnet is a representation of the "accessory genome" using a bipartite network. (see more in Lanza, V. F., Baquero, F., de la Cruz, F. & Coque, T. M. AcCNET (Accessory Genome Constellation Network): comparative genomics software for accessory genome analysis using bipartite networks. Bioinformatics 33, 283–285 (2017)). Accnet creates a graph using two kind of nodes: genomes and proteins/gene. One genome is connected to one proteins if the genome has this protein (i.e. a protein that belong to this protein family/cluster). The accnet object is other of the main data type in PATO. PATO can create some representations of the accessory genome (dendrograms, networks, plots....) and has some functions to analyze the accessory genome properly. The function accnet() builds an accnet object taking into account the maximun frequency of each protein/gene to be considered as part of accessory genome. Moreover, the user can decide to include single protein/genes or not (those which are only present in one sample).
Accnet depends of the definition of protein families so the input of the accnet() function is a mmseqs object. The definition of the homologous cluster is critical to define the accessory genome.
accnet() can be called as:
my_accnet <- accnet(my_mmseq, threshold = 0.8, singles = TRUE)
or piping the mmseqs() function
my_accnet <- mmseqs(my_files) %>% accnet(threshold = 0.8)
Workflow
This is an example analyzing 2.941 high quality Escherichia coli and Shigella downloaded from NCBI Assembly database (you can download a copy from https://doi.org/10.6084/m9.figshare.13049654). In this case we'll work with protein multi-fasta files (i.e. *.faa).
PATO can handle both nucleotide or aminoacid multi-fasta files. PATO does not use contigs fasta file. We recomend to use PROKKA output files .faa or .ffn. If you are using NCBI, EBI or other public database note that there are especific files for multi-fasta genes file or multi-fasta protein files. It is important to know that PATO take as annotation the headers of the fasta sequence, and usually, protein annotation is better than gen (CDS) annotation. Thus, we recommend to use protein fasta files.
It is very important to establish the working directory. We recommend to set the working directory in the folder where the multi-fasta files are. Make sure that you have enough free disk space.
library(pato) # We strongly recommend to load tidyverse meta-package library(tidyverse) setwd("/myFolder/")
The main input file for PATO is a plain-text list of files with the path to the multi-fasta files. This list could be easily made in your Unix console.
```{bash eval=FALSE, include = TRUE} ls *.faa > my_list.txt
... and import to R enviroment. ```r files <- read.table("my_list.txt",header = FALSE)
Quality Control
PATO has implemented a classification function to identify the most probable species determination for each input file. The method is based in the minimun mash distance of each file against a homemade reference database. Thr reference database comprises all reference and representative genomes of Refseq. Input files can be fasta nucleotides or aminoacids.
species <- classifier(files,n_cores = 20,type = 'prot') species %>% group_by(organism_name) %>% summarise(Number = n()) # A tibble: 8 x 2 organism_name Number <chr> <int> 1 Citrobacter gillenii 1 2 Escherichia coli O157:H7 str. Sakai 323 3 Escherichia coli str. K-12 substr. MG1655 2520 4 Escherichia marmotae 4 5 Shigella boydii 18 6 Shigella dysenteriae 7 7 Shigella flexneri 2a str. 301 45 8 Shigella sonnei 40
In this example we found 5 foreign genomes (1 Citrobacter and 4 E. marmotae). Now we can create a new list of files with the output of classifier. Using the tidyverse style we filter the unwanted sequences
files = species %>% filter(!grepl("Citrobacter",organism_name)) %>% filter(!grepl("marmotae",organism_name))
Now we create the main objects mashand mmseqs. We go to create a protein families with 80% of identity, 80% of coverage and a maximun of 1e-6 of Evalue.
ecoli_mash_all <- mash(files, n_cores = 20, type = 'prot') ecoli_mm<- mmseqs(files, coverage = 0.8, identity = 0.8, evalue = 1e-6, n_cores = 20)
In my case, I have 20 cores in my computer.
Finally we can create an accnet object:
ecoli_accnet_all <- accnet(ecoli_mm,threshold = 0.8, singles = FALSE)
Pre-processing
Outliers
One of the things that happen when you download genomes from public databases is that some times you can find some outliers, genomes that not belong to the selected specie. PATO implement a function to indentify outliers and to remove it if is neccessary. In this case we set a threshold of 0.06 (~94% of ANI) to be considered as outlier.
outl <- outliers(ecoli_mash_all,threshold = 0.06)
You can remove the outliers from mash or accnet objects
ecoli_mash_all <-remove_outliers(ecoli_mash_all, outl) ecoli_accnet_all <-remove_outliers(ecoli_accnet_all, outl)
(see )
If you desire to remove outliers from files object you can do it
files <- anti_join(files, outl, by=c("V1"="Source"))
This command remove the outliers from the original files object. We recommend do it before mmseqs command.
Redundancies
Most of the times in the public databases are redundancies. The inclusion of outbreaks and studies about specific Sequence Types or Sub-clones in the databases bias the real diversity of the species. PATO implement a function to select a non-redundant subset based on a fix distance (similarity), a % of samples or a selected number of genomes. It is an iterative approach so we implement a threshold of toerance due to some time find the exact solution is impossible. This function accepts mash or accnet object as input. Among, there is a function to create mash or accnet objects extracting the subset of genomes
# For select 800 non redundant samples nr_list <- non_redundant(ecoli_mash_all,number = 800)
or
# For select a subset of samples with 99.99% of indentity nr_list <- non_redundant(ecoli_mash_all, distance = 0.0001)
to create the objects only with the representatives of each cluster:
ecoli_accnet <- extract_non_redundant(ecoli_accnet_all, nr_list) ecoli_mash <- extract_non_redundant(ecoli_mash_all, nr_list)
Moreover, there is another function non_redundant_hier() that perform a hierarchical approach to perform the clusterization of the genomes. It is faster for big dataset (>1000 genomes) so we recommend use it in this cases.
nr_list <- non_redundant_hier(ecoli_mash_all,800, partitions = 10) ecoli_accnet <- extract_non_redundant(ecoli_accnet_all, nr_list) ecoli_mash <- extract_non_redundant(ecoli_mash_all, nr_list)
The partition parameter set the number of hierarchical steps.
Core Genome analysis
PATO implements a set of tools to inspect the core genome. It includes the pangenome composition (core, accessory, pangenome) and the function to create a core-genome alignment.
We can inspect the Pangenome composition of our dataset with:
cp <- core_plots(ecoli_mm,reps = 10, threshold = 0.95, steps = 10)
PATO can build a core-genome alignment to use with external software such as IQTree[http://www.iqtree.org/], RAxML-NG[http://www.exelixis-lab.org/software.html], FastTree[http://www.microbesonline.org/fasttree/] or other phylogenetic inference software. The core genome is computed based on a mmseq object, so the definition of core-genome depends of the parameters used in that step.
In this example we build a core-genome aligment of the ~800 samples of non-redundant result.
nr_files = nr_list %>% as.data.frame() %>% group_by(cluster) %>% top_n(1,centrality) %>% summarise_all(first) %>% select(Source) %>% distinct()
We have selected the best sample of each cluster (max centrality value). Some cluster has several samples with the same centrality value, so we take one of them (the first one). Now we run the core genome for this list of samples.
core <- mmseqs(nr_files) %>% core_genome(type = "prot") export_core_to_fasta(core,"core.aln")
Some time, when you are using public data, core-genome can be smaller than you expect. Citating Andrew Page, creator of Roary (https://sanger-pathogens.github.io/Roary/)
I downloaded a load of random assemblies from GenBank. Why am I seeing crazy results?
Gene prediction software rarely completely agrees, with differing opinions on the actual start of a gene, or which of the overlapping open reading frames is actually the correct one, etc. As a result, if you mix these together you can get crazy results. The solution is to reannotate all of the genomes with a single method (like PROKKA). Otherwise you will waste vast amounts of time looking at noise, errors, and the batch effect.
other times you can have a problem of outliers
The exported files are in Multi-alignment FASTA format and ca be use with most of the Phylogenetic tools. In this case we used FastTree for phylogenetic inference
```{bash eval=FALSE, include=TRUE} fasttreeMP core.aln > core.tree
And then we can read the output file to import to our R environment and plot the result. ```r library(phangorn) library(ggtree) core_tree = read.tree("core.tree") annot_tree = species %>% filter(!grepl("Citrobacter",organism_name)) %>% filter(!grepl("marmotae",organism_name)) %>% select(Target,organism_name) %>% distinct() core_tree %>% midpoint %>% ggtree(layout = "circular") %<+% annot_tree + geom_tippoint(aes(color = organism_name))
In this case we have added the species information extracted above.
You must take into account that Maximun Likelyhood tree can take long computational times.
Finally, you can obtain a SNPs matrix from the coregenome alignment. The matrix shown the number of total SNPs (or changes in the case of proteins alignments) shared among the samples. The matrix can be normalized by the total length of the genome (in magabases) or can be raw numer of variants.
var_matrix = core_snps_matrix(core, norm = TRUE) pheatmap::pheatmap(var_matrix,show_rownames = F, show_colnames = F)
Accessory Genome Analysis
I this case we are creating the accessory genome taking those proteins presence in no more than 80% of the genomes.
ecoli_accnet_all <- accnet(ecoli_mm,threshold = 0.8, singles = FALSE)
As we have shown above, PATO has an object type to accessory genome accnet and functions to analyze and visualize the content of the accessory genome and the relationship between genes/proteins and the genomes. We can visualize an accnet object as a bipartite network. Commonly, AccNET networks are very large, so, we recommend to visualize the networks using Gephi. We do not recommend to try visualize AccNET networks with more than 1.000 genomes.
One of the most interesting things when we analyze an accessory genome is try to find what genes/proteins are over-represented in a set of genomes. accnet_enrichment_analysis() function analyze what genes/proteins are over-represented in a cluster in comparisson with the population (i.e. in the wholem dataset). The clusters definition can be external (any catagorical metadata such as Sequence Type, Source, Serotype etc...) or internal using some clustering proccess. We have to take into account, that the redundancies can bias this kind of analysis. If there are an over-representation of samples, for example an outbreak, in your dataset the results could be bias. You will find more significant genes/proteins because the diversity of the dataset is not homegenously distributed. For this reason we recommend to use a non redundant set of samples. In this example we select 800 non redundant genomes.
ec_nr<- non_redundant(ecoli_mash,number = 800 ) ecoli_accnet_nr <- extract_non_redundant(ecoli_accnet, ef_nr) ec_800_cl <- clustering(ecoli_accnet_nr, method = "mclust", d_reduction = TRUE)
Now we can visualize the network using Gephi.
export_to_gephi(ecoli_accnet_nr, "accnet800", cluster = ec_800_cl)
[https://doi.org/10.6084/m9.figshare.13089338]
To perform the enrichment analysis we use:
accnet_enr_result <- accnet_enrichment_analysis(ecoli_accnet_nr, cluster = ef_800_cl) # A tibble: 1,149,041 x 14 Target Source Cluster perClusterFreq ClusterFreq ClusterGenomeSize perTotalFreq TotalFreq OdsRatio pvalue padj AccnetGenomeSize AccnetProteinSi… Annot <chr> <chr> <dbl> <dbl> <int> <int> <dbl> <int> <dbl> <dbl> <dbl> <int> <int> <chr> 1 1016|NZ_CP053592.… GCF_000009565.1_ASM956v… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 2 1016|NZ_CP053592.… GCF_000022665.1_ASM2266… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 3 1016|NZ_CP053592.… GCF_000023665.1_ASM2366… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 4 1016|NZ_CP053592.… GCF_000830035.1_ASM8300… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 5 1016|NZ_CP053592.… GCF_000833145.1_ASM8331… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 6 1016|NZ_CP053592.… GCF_001039415.1_ASM1039… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 7 1016|NZ_CP053592.… GCF_001596115.1_ASM1596… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 8 1016|NZ_CP053592.… GCF_009663855.1_ASM9663… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 9 1016|NZ_CP053592.… GCF_009832985.1_ASM9832… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" 10 1016|NZ_CP053592.… GCF_013166955.1_ASM1316… 1 0.0436 13 298 0.0173 20 2.52 3.62e-5 0.00130 1156 74671 "" # … with 1,149,031 more rows
[https://doi.org/10.6084/m9.figshare.13089314.v2]
Now, we can export a new network with the adjusted p-values as edge-weigth.
accnet_with_padj(accnet_enr_result) %>% export_to_gephi("accnet800.padj", cluster = ec_800_cl)
[https://doi.org/10.6084/m9.figshare.13089401.v1]
PATO also include some functions to study the genes/proteins distribution: singles(), twins() and conicidents().
singles()finds those genes/proteins that are only present in a sample (genome, pangenome...).twins()finds those genes/proteins that have the same connections (i.e. genes/proteins present in the same genomes).coincidents()finds those genes/proteins with similar connections (i.e. genes/proteins that usually are together)
The differencies between twins() and concidents() are that coincidents() is more flexible and therefore less sensitive to outliers. twins() is faster and accurate but is very sensitive to noise or outliers because just one missing conection (for example a bad prediction of a protein in a genome) remove automatically that proteins from the twin group. On other hand, coincidents() is slower, and some time the results are to much generalistic giving big cluster of pseudo-core.
Population structure
PATO includes tools for population structure search and comparisson. PATO can analyse the population structure of the whole-genome (MASH based) or the accessory structure (AccNET based)
We can visualize our dataset as a dendrogram (i.e a tree). PATO allows to visualize both mash and accnet data as a tree. In the caso of accnet data, first PATO calculate a distance matrix using the presence/absence matrix in combination with Jaccard distance. The function similarity_tree() implement different methods to build the dendrogram. Phylogenetic aproaches such as Neighbour Joining, FastME:Minimun Evolution and hierarchical clustering method such us complete linkage, UPGMA, Ward's mininum variance, and WPGMC.
mash_tree_fastme <- similarity_tree(ecoli_mash) mash_tree_NJ <- similarity_tree(ecoli_mash, method = "NJ") mash_tree_upgma <- similarity_tree(ecoli_mash,method = "UPGMA") accnet_tree_upgma <- similarity_tree(ecoli_accnet,method = "UPGMA")
The output has a phylo format, so can be visualize with external packages as ggtree.
mash_tree_fastme %>% midpoint %>% ggtree(layout = "circular") %<+% annot_tree + geom_tippoint(aes(color = organism_name))
Using others external packages, we can compare the arragement of the pangenom (mash data) against the accessory genome (accnet)
library(dendextend) tanglegram(ladderize(mash_tree_upgma), ladderize(accnet_tree_upgma), fast = TRUE, main = "Mash vs AccNET UPGMA")
Some Maximun Likelyhood inference trees software accept,as input, binary data (0-1) alignments.So, we can use accessory data (accnet) to infere a tree (non-phylogenetic) with this data. This is similar to the similarity_tree() but instead to be based on distance metrics it's based on ML principles. To export this alignment you can use export_accnet_aln().
mmseqs(files) %>% accnet() %>% export_accnet_aln(.,file ="my_accnet_aln.fasta")
And then you can use it as input alignment
```{bash eval=FALSE, include=FALSE}
iqtree -s my_accnet_aln.fasta -st BIN
Again, we can import to R and plot it. ```r acc_tree = read.tree("acc.aln.treefile") acc_tree %>% midpoint.root() %>% ggtree()
This kind of alignments can be very large. Most of the times accessory genomes contains a lot of spurious genes/proteins that do not add information to the alignment. For this reason export_accnet_aln() has a parameter, min_freq, to filter the genes/proteins by their frequency. This option cut significantly the alingment length and improves computational times.
PATO has a set of function to visualize data. We have saw the tree but algo implements methods to visualize as networks the relationships among genomes. K-Nearest Neighbour Networks is a representation of the realtionship of the data in a network plot. This function (knnn()) builds a network linking each genome with their K best neighbours instead of chose a distance threshold. This approach minimize the clutering of the network and tries to create a connected network. The method can chose the K best neighbours with or without repetitions. That means that if repeats=FALSE each node are linked with the best K neighbours that does be not linked yet with the node.
# K-NNN with 10 neighbours and with repetitions knnn_mash_10_w_r <- knnn(ecoli_mash,n=10, repeats = TRUE) # K-NNN with 25 neighbours and with repetitions knnn_mash_25_w_r <- knnn(ecoli_mash,n=25, repeats = TRUE) # K-NNN with 50 neighbours and with repetitions knnn_mash_50_w_r <- knnn(ecoli_mash,n=50, repeats = TRUE)
knnn() function returns an igraph object that can be visualize with several packages. However, PATO implements its own function plot_knnn_network() that uses threejs and the own igraph for layouting and plotting the network. Due to the common size of the network we strongly recommend to use this function or use the external software Gephi (https://gephi.org/). You can use the function export_to_gephi() to export these networks to Gephi.
export_to_gephi(knnn_mash_50_w_r,file = "knnn_50_w_r.tsv")
If you want, you can use the internal function plot_knnn_network() to visualize the network. This funtion uses igraph layouts algorith to arrange the network and threejs package to draw and explore the network.
plot_knnn_network(knnn_mash_50_w_r)
PATO includes a different clustering methods for different kind of data (objects). clustering() function joins all the different approaches to make easier the process. For mash and accnetobjects we have the following methods:
- mclust: It perform clustering using Gaussian Finite Mixture Models. It could be combine with d_reduction. This method uses Mclust package. It has been implemented to find the optimal cluster number
- upgma: It perform a Hierarchical Clustering using UPGMA algorithm. The user must provide the number of cluster.
- ward.D2: It perform a Hierarchical Clustering using Ward algorithm. The user must provide the number of cluster.
- hdbscan: It perform a Density-based spatial clustering of applications with noise using DBSCAN package. It find the optimal number of cluster.
Any of the above methods is compatible with a multidimentsional scaling (MDS). PATO performs the MDS using UMAP algorithm (Uniform Manifold Approximation and Projection). UMAP is a tools for MDS, similar to other machine learning MDS technics as t-SNE or Isomap. Dimension reduction algorithms tend to fall into two categories those that seek to preserve the distance structure within the data and those that favor the preservation of local distances over global distance. Algorithms such as PCA , MDS , and Sammon mapping fall into the former category while t-SNE, Isomap or UMAP,fall into the later category. This kind of MDS combine better with the clustering algorithms due to clustering process try to find local structures.
ec_cl_mclust_umap <- clustering(ecoli_mash, method = "mclust",d_reduction = TRUE)
On other hand, clustering() can handle knnn networks as input. In this caso, PATO uses network clustering algorithms such us:
- greedy: Community structure via greedy optimization of modularity
- louvain: This method implements the multi-level modularity optimization algorithm for finding community structure
- walktrap: Community strucure via short random walks
ec_cl_knnn <-clustering(knnn_mash_50_w_r, method = "louvain")
Whatever the method used, the output has allways the same structure: a data.frame with two columns c("Source","Cluster"). The reason of this format is to be compatible with the rest of the data, being able to combine with the rest of the objects using Source variable as key.
To visualize the clustering data or just to see the data structure we can use umap_plot() function. The functions performs a umap reduction and plot the results. We can include the clustering results as a parameter and visualize it.
umap_mash <- umap_plot(ecoli_mash)
umap_mash <- umap_plot(ecoli_mash, cluster = ef_cl_mclust_umap)
We also can use plot_knnn_network() to visualize the network clustering results.
cl_louvain = clustering(knnn_mash_25_wo_r, method = "louvain") plot_knnn_network(knnn_mash_25_wo_r, cluster = cl_louvain, edge.alpha = 0.1)
Annotation
PATO has a function to annotate the Antibiotic Resistance Genes and the Virulence Factor: annotate(). Antibiotic resistance are predicted using MMSeqs2 over ResFinder database (https://cge.cbs.dtu.dk/services/ResFinder/) (doi: 10.1093/jac/dks261). For Virulence Factors we use VFDB (http://www.mgc.ac.cn/cgi-bin/VFs/v5/main.cgi)(doi: 10.1093/nar/gky1080). VFBD has two sets of genes VF_A the core-dataset and VF_B the full dataset. Core dataset includes genes associated with experimentally verified VFs only, whereas full dataset covers all genes related to known and predicted VFs in the database.
annotate() results is a table with all positive hits of each gene/protein of the dataset (files). annotate() can re-use the results of mmseqs() to accelerate the proccess. In the same way, the query can be all genes/proteins or accessory genes/proteins. The results are quite raw so user must curate the table. We recommend to use the tidyverse tools. It is very easy to obtain a greatfull results.
library(tidyverse) annotation <- annotate(files, type = "prot",database = c("AbR","VF_A")) annotation %>% filter(pident > 0.95 ) %>% #remove all hits with identity lower than 95% filter(evalue < 1e-6) %>% #remove all hits with E-Value greater than 1e-6 group_by(Genome,Protein) %>% arrange(desc(bits)) %>% slice_head() #select only the best hit for each protein-genome.
PATO includes also a function to create a heapmap with the annotation:
heatmap_of_annotation(annotation %>% filter(DataBase =="AbR"), #We select only "AbR" results min_identity = 0.99)
Or to visualizae as a network.
network_of_annotation(annotation %>% filter(DataBase =="AbR"), min_identity = 0.99) %>% export_to_gephi("annotation_Network")
[https://doi.org/10.6084/m9.figshare.13121795]
Pangenomes Analysis
PATO is designed even to analyze the relationship among pangenomes of different (or not) species. Thats mean to analyze cluster of genomes (pangenomes) among then as individual elements. In this example we analyze 49.591 firmicutes genomes from NCBI data base. For that kind of large datasets it is recommended to use the specific pipeline for pangenomes. This pipeline first cluster the genomes into pangenomes, for example, cluster of species or intra-specie phylogroups or even sequence types (ST). The diversity of each cluster depends on the parameter distance. The pipeline creates homogeneous (in phylogenetic distance) clusters. Then the pipeline produces a accnet object so all the above described function can be use with this object as a common accnet object. The pipeline also include parameters to set the minimum amount of genomes to consider a pangenome and the minimum frequency of a protein/gene family to be included in a pangenome.
res <- pangenomes_from_files(files,distance = 0.03,min_pange_size = 10,min_prot_freq = 2) export_to_gephi(res,"/storage/tryPATO/firmicutes/pangenomes_gephi")
In this case we produce a gephi table to visulize the accnet network of the dataset. To annotate the network we use the NCBI assembly table and takes the species name of each pangenome cluster.
assembly = data.table::fread("ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/assembly_summary.txt",sep = "\t",skip = 1, quote = "") colnames(assembly) = gsub("# ","",colnames(assembly)) annot = res$members %>% mutate(file = basename(path)) %>% separate(file,c("GCF","acc","acc2"),sep = "_", remove = FALSE) %>% unite(assembly_accession,GCF,acc,sep = "_") %>% left_join(assembly) %>% separate(organism_name,c("genus","specie"), sep = " ") %>% group_by(pangenome,genus,specie) %>% summarise(N = n()) %>% distinct() %>% group_by(pangenome) %>% top_n(1,N) %>% mutate(ID = paste("pangenome_",pangenome,"_rep_seq.fasta", sep = "",collapse = "")) %>% write_delim("/storage/tryPATO/firmicutes/pangenomes_gephi_extra_annot.tsv",delim = "\t", col_names = TRUE)
We chose as species name the specie most frequent in each cluster. Then usin Gephi we visualiza the network.
{width=100%}
Other way to visualize the results is a heatmap of the sharedness among the pangenoms. PATO includes the function sharedness to visualize this result. To visualize the data.table result from sharedness() function we can use the package pheatmap. As the input value of pheatmap can be only a matrix we must first transform our result into a matrix:
sh <- sharedness(res) sh <- sh %>% as.data.frame() %>% rownames_to_column("ID") %>% inner_join(annot) %>% unite(Name,genus,specie,pangenome, sep = "_") %>% select(-ID,-N)%>% column_to_rownames("Name") colnames(sh) = rownames(sh)
Now we can execute pheatmap() with our pangenome dataset.
pheatmap::pheatmap(sh,clustering_method = "ward.D2", clustering_distance_cols = "correlation", clustering_distance_rows = "correlation")
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.