Nothing
CNE identification and visualisation
In CNEr: CNE Detection and Visualization
date = "`r doc_date()`"
pkg = "`r pkg_ver('CNEr')`"
short=TRUE #if short==TRUE, do not echo code chunks
debug=FALSE
knitr::opts_chunk$set(echo=!short, warning=debug, message=debug, error=FALSE,
cache.path = "cache/",
fig.path = "figures/")
Introduction
Conserved noncoding elements (CNEs) are a pervasive class of elements
clustering around genes with roles in development and differentiation
in Metazoa [@Woolfe:2004ur].
While many have been shown to act as long-range developmental enhancers
[@Sandelin:2004bd],
the source of their extreme conservation remains unexplained.
To study the evolutionary dynamics of these elements
and their relationship to the genes around which they cluster,
it is essential to be able to produce genome-wide sets of these elements
for a large number of species comparisons,
each with multiple size and conservation thresholds.
This r Biocpkg("CNEr") package aims to detect CNEs and
visualise them along the genome.
For performance reasons, the implementation of CNEs detection
and corresponding I/O functions are primarily written as C extensions to R.
We have used r Biocpkg("CNEr") to produce sets of CNEs by scanning pairwise whole-genome net alignments
with multiple reference species,
each with two different window sizes and a range of minimum identity thresholds.
Then, to pinpoint the boundaries of CNE arrays,
we compute the CNE densities as the percentages of length
covered by CNEs within a user specified window size.
Finally, we describe a novel visualisation method using horizon plot tracks
that shows a superior dynamic range to the standard density plots,
simultaneously revealing CNE clusters characterized
by vastly different levels of sequence conservation.
Such CNE density plots generated using precise locations of CNEs
can be used to identify genes involved in developmental regulation,
even novel genes that are not annotated yet.
Workflow of the package
This section briefly demonstrates the pipeline of CNE identification
and visualisation.
More detailed usage of each step is described in following sections with a concise example of CNE identification and visualisation for the "barhl2" and "sox14" (chr6:24,000,000..27,000,000) loci in Zebrafish (danRer10) genome against Human (hg38).
CNE identification
- Preparation of axtNets: the axtNet files can be downloaded from UCSC
or generated by this package.
- Identification of conserved noncoding regions: scan the axt alignments for the regions, with minimal I identities over C columns, which do not overlap with annotated exons/repeats.
- Merging overlapped elements : scanned elements from two pairs of alignments (one from each genome as reference) that overlap on both genomes are merged.
- Removal of unannotated repetitive sequences: realign the elements back to respective genome with BLAT and discard the elements which exceed a certain number of hits on either genome.
CNE visualisation
- Display parameters: chromosome, start, end, smooth window size.
- Horizon plot: visualise CNE densities with the package
r Biocpkg("Gviz").
Input
The minimal input for r Biocpkg("CNEr") includes the whole genome pairwise alignment of two assemblies, axt
net files.
UCSC already provides a set of precomputed axt files on http://hgdownload.soe.ucsc.edu/downloads.html for most of popular genomes.
In case the axt net files are not available from UCSC, you can always generate the axt net files by following another vignette "Pairwise whole genome alignment" in this r Biocpkg("CNEr") package.
Another essential information is the annotation of exons and repeats, which could be retrieved from various resources.
axt alignment file
During the development of this package, there was no suitable class to store the axt alignments in Bioconductor.
Hence, we created two new S4 classes, Axt and GRangePairs, to easily manipulate the axt alignment files.
GRangePairs is designed to hold a pair of GRanges objects, which have the same length.
It builds on the Pairs class of Bioconductor and inherits many useful methods from it.
This Axt class inherits from GRangePairs with no extra slots to hold the content from axt files, but many specific methods are created for Axt.
The ranges of the target and the query organism are stored in two GRanges objects with alignment sequences as metadata columns.
The Blastz scores and the widths of the alignments are stored in metadata columns of GRangePairs.
For more information on the usage of these two classes, please refer to the documentation.
To read axt file into R,
r Biocpkg("CNEr") provides readAxt function for highly efficient reading.
This function is built on a backend C code of Kent's utilities [@Kent:2002bw].
The axt alignment files can be either gzippped or in plain text file.
The alignments between two genomes can also be in one big file or in several files, such as "chr1.hg19.mm10.net.axt.gz", "chr2.hg19.mm10.net.axt.gz", etc.
library(CNEr)
## These axt files are specially prepared for the region
## (chr6:24,000,000..27,000,000)
axtFilesHg38DanRer10 <- file.path(system.file("extdata", package="CNEr"),
"hg38.danRer10.net.axt")
axtFilesDanRer10Hg38 <- file.path(system.file("extdata", package="CNEr"),
"danRer10.hg38.net.axt")
axtHg38DanRer10 <- readAxt(axtFilesHg38DanRer10)
axtDanRer10Hg38 <- readAxt(axtFilesDanRer10Hg38)
## Axt class is shown in UCSC axt format
axtHg38DanRer10
axtDanRer10Hg38
## Distribution of matched alignments; Given an Axt alignment, plot a heatmap with percentage of each matched alignment
matchDistribution(axtHg38DanRer10)
matchDistribution(axtDanRer10Hg38)
## Example of chr4 on hg19 and galGal3
## The synteny of human and zebrafish is not quite obvious on the dotplot.
library(BSgenome.Hsapiens.UCSC.hg19)
library(BSgenome.Ggallus.UCSC.galGal3)
fn <- file.path(system.file("extdata", package="CNEr"),
"chr4.hg19.galGal3.net.axt.gz")
axt <- readAxt(fn,
tAssemblyFn=file.path(system.file("extdata",
package="BSgenome.Hsapiens.UCSC.hg19"),
"single_sequences.2bit"),
qAssemblyFn=file.path(system.file("extdata",
package="BSgenome.Ggallus.UCSC.galGal3"),
"single_sequences.2bit"))
library(GenomeInfoDb)
syntenicDotplot(axt, firstChrs=c("chr4"), secondChrs="chr4", type="dot")
There are methods defined for handling Axt objects, including subsetting, output to axt files.
More details can be found in the man page.
Filtering information
The gene annotation information, including exons and repeats, is used to filter out the undesired regions.
Here we summarise a table of filtering information we used:
Assembly | Name | Exon | Repeat
----------- | ----------- | ----------------------------------------- | ---------
hg38 | Human | RefSeq Genes, Ensembl Genes, UCSC Known Genes | RepeatMasker
mm10 | Mouse | RefSeq Genes, Ensembl Genes, UCSC Known Genes | RepeatMasker
xenTro3 | Frog | RefSeq Genes, Ensembl Genes | RepeatMasker
tetNig2 | Tetraodon | Ensembl Genes | |
canFam3 | Dog | RefSeq Genes, Ensembl Genes | RepeatMasker
galGal4 | Chicken | RefSeq Genes, Ensembl Genes | RepeatMasker
danRer10 | Zebrafish | RefSeq Genes, Ensembl Genes | RepeatMasker
fr3 | Fugu | RefSeq Genes | RepeatMasker
anoCar2 | Lizard | Ensembl Genes | RepeatMasker
equCab2 | Horse | RefSeq Genes, Ensembl Genes | RepeatMasker
oryLat2 | Medaka | RefSeq Genes, Ensembl Genes | RepeatMasker
monDom5 | Opossum | RefSeq Genes, Ensembl Genes | RepeatMasker
gasAcu1 | Stickleback | RefSeq Genes, Ensembl Genes | RepeatMasker
rn5 | Rat | RefSeq Genes, Ensembl Genes | RepeatMasker
dm3 | D. melanogaster | RefSeq Genes, Ensembl Genes | RepeatMasker
droAna2 | D. ananassae | | RepeatMasker
dp3 | D. pseudoobscura | | RepeatMasker
ce4 | C. elegans | RefSeq Genes | RepeatMasker
cb3 | C. briggsae | | RepeatMasker
caeRem2 | C. remanei | | RepeatMasker
caePb1 | C. brenneri | | RepeatMasker
For the sake of simplicity, all the information listed above can be fetched easily
with Bioconductor package r Biocpkg("rtracklayer"), r Biocpkg("biomaRt") and
precompiled Bioconductor annotation packages.
A few examples are given here:
## To fetch rmsk table from UCSC
library(rtracklayer)
mySession <- browserSession("UCSC")
genome(mySession) <- "hg38"
hg38.rmsk <- getTable(ucscTableQuery(mySession, track="RepeatMasker",
table="rmsk"))
hg38.rmskGRanges <- GRanges(seqnames=hg38.rmsk$genoName,
## The UCSC coordinate is 0-based.
ranges=IRanges(start=hg38.rmsk$genoStart+1,
end=hg38.rmsk$genoEnd),
strand=hg38.rmsk$strand)
## To fetch ensembl gene exons from BioMart
library(biomaRt)
ensembl <- useMart(biomart="ENSEMBL_MART_ENSEMBL",
host="dec2015.archive.ensembl.org")
ensembl <- useDataset("hsapiens_gene_ensembl",mart=ensembl)
attributes <- listAttributes(ensembl)
exons <- getBM(attributes=c("chromosome_name", "exon_chrom_start",
"exon_chrom_end", "strand"),
mart=ensembl)
exonsRanges <- GRanges(seqnames=exons$chromosome_name,
ranges=IRanges(start=exons$exon_chrom_start,
end=exons$exon_chrom_end),
strand=ifelse(exons$strand==1L, "+", "-")
)
seqlevelsStyle(exonsRanges) <- "UCSC"
## Use the existing Bioconductor annotation package for hg38
library(TxDb.Hsapiens.UCSC.hg38.knownGene)
exonsRanges <- exons(TxDb.Hsapiens.UCSC.hg38.knownGene)
The regions to filter out can also be provided in a bed file.
To import the bed file into r Biocpkg("GRanges") in R,
r Biocpkg("rtracklayer") provides a general function
import.bed to do that.
Since only the chromosome names, start and end coordinates are used in r Biocpkg("CNEr"),
we provide a more efficient readBed function.
## Existing bed file for chr6:24,000,000..27,000,000 of Zebrafish danRer10
bedDanRer10Fn <- file.path(system.file("extdata", package="CNEr"),
"filter_regions.danRer10.bed")
danRer10Filter <- readBed(bedDanRer10Fn)
danRer10Filter
## Existing bed file for alignment region in Human hg38 against
## chr6:24,000,000..27,000,000 of danRer10
bedHg38Fn <- file.path(system.file("extdata", package="CNEr"),
"filter_regions.hg38.bed")
hg38Filter <- readBed(bedHg38Fn)
hg38Filter
Creating a CNE class
We designed a CNE class to store all metadata of running the pipeline for identifying a set of CNEs between two species, including the intermediate and final results.
CNE can be created by providing the paths of the twoBit files of two assemblies, and
the paths of axt files, with each assembly as reference.
## Here we have the twoBit files from Bioconductor package
## BSgenome.Drerio.UCSC.danRer10 and BSgenome.Hsapiens.UCSC.hg38
cneDanRer10Hg38 <- CNE(
assembly1Fn=file.path(system.file("extdata",
package="BSgenome.Drerio.UCSC.danRer10"),
"single_sequences.2bit"),
assembly2Fn=file.path(system.file("extdata",
package="BSgenome.Hsapiens.UCSC.hg38"),
"single_sequences.2bit"),
axt12Fn=axtFilesDanRer10Hg38, axt21Fn=axtFilesHg38DanRer10,
cutoffs1=8L, cutoffs2=4L)
cneDanRer10Hg38
Note: the order of assemblies when creating CNE object is important.
Here we have danRer10 as assembly1 and hg38 as assembly2.
Then the axt12Fn contains the axt alignment with assembly1 danRer10 as reference and axt21Fn contains the alignment with assembly2 hg38 as reference.
The cutoffs1 and cutoffs2 are the maximal number of hits during the realignment in later steps.
Because zebrafish has undergone additional whole genome duplication compared to human, the cutoffs of zebrafish also doubles the cutoffs of human.
CNE identification
In this section, we will go through the details of CNE identification.
Scan axt alignments
Detecting CNEs highly relies on the whole-genome pairwise net alignments.
To correct the bias of a chosen genome (which bias?) and
capture the duplicated CNEs during genome evolution,
we scan two sets of nets for each pairwise comparison,
one as reference from each of the genomes.
We identify CNEs by scanning the alignments for regions with
at least I identities over C alignment columns.
Because different genes and loci may favor various similarity scores,
we usually scan at two diffrent window sizes 30 and 50 with
several similarity criterias (I/C) range from 70% to 100%.
identities <- c(45L, 48L, 49L)
windows <- c(50L, 50L, 50L)
## Here danRer10Filter is tFilter since danRer10 is assembly1
cneListDanRer10Hg38 <- ceScan(x=cneDanRer10Hg38, tFilter=danRer10Filter,
qFilter=hg38Filter,
window=windows, identity=identities)
At this stage, a list of CNE is returned from ceScan, which contains the preliminary two sets of CNEs from a pair of axt alignments.
We can examine the intermediate CNEs by
## CNEs from the alignments with danRer10 as reference
CNE12(cneListDanRer10Hg38[["45_50"]])
## CNEs from the alignments with hg38 as reference
CNE21(cneListDanRer10Hg38[["45_50"]])
In the result table, even though the strand for query element can be negative,
the coordinate for that query element is already on the positive strand.
It is essential to scan two sets of pairwise net alignments with each assembly as reference, in order not to miss any duplicated elements in either lineage.
This is particularly important for the comparison between teleost fishes and other vertebrates, because one (for instance, the case of zebrafish) or two (the case of common carp) extra whole genome duplications occured.
Merge CNEs
As we perform two rounds of CNE detection with each genome as reference,
some conserved elements overlap on both genomes and should be removed.
Elements, however, that overlap only on one of the genomes are kept,
so that duplicated elements remain distinct.
cneMergedListDanRer10Hg38 <- lapply(cneListDanRer10Hg38, cneMerge)
Realignment of CNEs
Some CNEs might be unannotated repeats.
To remove them, currently we use blat [@Kent:2002jd] to realign
each sequence of CNEs against the respective genomes.
When the number of matches exceeds a certain threshold, for instance 8,
that CNE will be discarded.
This step can be very time-consuming when the number of CNEs is large.
Other alignment methods could also be considered, for example Bowtie2 or BWA (provided that they are installed on the user's machine)
cneFinalListDanRer10Hg38 <- lapply(cneMergedListDanRer10Hg38, blatCNE)
CNE storage and query
As the computation of CNEs from the whole pipeline and
the preparation of annotation package can be very time-consuming,
for a smoother visualisation experience,
we decided to use a local SQLite database in order to store CNEs.
Since the CNEs data.frame is a table, it can be imported into a SQL table.
To speed up the query from the SQL database,
the bin indexing system is adopted.
For more information, please refer to the paper [@Kent:2002bw]
and genomewiki.
## on individual tables
dbName <- tempfile()
data(cneFinalListDanRer10Hg38)
tableNames <- paste("danRer10", "hg38", names(cneFinalListDanRer10Hg38),
sep="_")
for(i in 1:length(cneFinalListDanRer10Hg38)){
saveCNEToSQLite(cneFinalListDanRer10Hg38[[i]], dbName, tableNames[i],
overwrite=TRUE)
}
When querying results from the local SQLite database based on the chr,
coordinates and other criterias,
a GRanges object is returned.
chr <- "chr6"
start <- 24000000L
end <- 27000000L
minLength <- 50L
tableName <- "danRer10_hg38_45_50"
fetchedCNERanges <- readCNERangesFromSQLite(dbName, tableName, chr,
start, end, whichAssembly="first",
minLength=minLength)
fetchedCNERanges
CNE length distribution
As the lengths of CNEs [@Salerno:2006sc] and the distances between consecutive elements [@Polychronopoulos:2014pl] exhibit power-law distributions, we implemented a function that might be useful in showing interesting patterns in the distribution of the identified elements.
dbName <- file.path(system.file("extdata", package="CNEr"),
"danRer10CNE.sqlite")
tAssemblyFn <- file.path(system.file("extdata",
package="BSgenome.Drerio.UCSC.danRer10"),
"single_sequences.2bit")
qAssemblyFn <- file.path(system.file("extdata",
package="BSgenome.Hsapiens.UCSC.hg38"),
"single_sequences.2bit")
cneGRangePairs <- readCNERangesFromSQLite(dbName=dbName,
tableName="danRer10_hg38_45_50",
tAssemblyFn=tAssemblyFn,
qAssemblyFn=qAssemblyFn)
plotCNEWidth(cneGRangePairs)
Genomic distribution of CNEs along the chromosome
CNEs tend to form clusters. A quick check of the genomic distribution of CNEs
is available.
plotCNEDistribution(first(cneGRangePairs))
Output of bed and bedGraph files
For visualisation in other Genome Browser, we provide functions to generate the CNE in bed files and CNE density in bedGraph files.
For example, to get the first 1000 coordinates of CNEs:
makeCNEDensity(cneGRangePairs[1:1000])
CNEs visualisation
To visualise CNEs alongside other gene annotations,
we choose to use the Bioconductor package r Biocpkg("Gviz") in this vignette.
r Biocpkg("Gviz"), based on the r CRANpkg("grid") graphics scheme,
is a very powerful package for plotting data and annotation information
along genomic coordinates.
The functionality of integrating publicly available genome annotation data,
such as UCSC or Ensembl,
significantly reduced the burden of preparing annotations for common assemblies.
Since the Bioconductor release 2.13 of r Biocpkg("Gviz"),
it provides the data track in horizon plot,
which exactly meets our needs for visualisation of CNEs density plots.
For more detailed usage, please check the vignette or manual of r Biocpkg("Gviz").
Another option for visualisation is the package r Biocpkg("ggbio"),
which is based on r CRANpkg("ggplot2").
The advantage of r Biocpkg("ggbio") is the simplicity of
adding any customised r CRANpkg("ggplot2") style track
into the plot without tuning the coordinate systems.
The densities generated in the following section can be easily plot in the
horizon plot.
A short straightforward tutorial regarding horizon plot
in ggplot2 format
is available from
http://timelyportfolio.blogspot.co.uk/2012/08/horizon-on-ggplot2.html.
Gene annotation visualisation
For the example case of hg38 vs danRer10 in this vignette,
we choose danRer10 as the reference and
show the range of developmental gene barhl2 and sox14.
library(Gviz)
library(biomaRt)
genome <- "danRer10"
axisTrack <- GenomeAxisTrack()
cpgIslands <- UcscTrack(genome=genome, chromosome=chr,
track="cpgIslandExt", from=start, to=end,
trackType="AnnotationTrack", start="chromStart",
end="chromEnd", id="name", shape="box",
showId=FALSE,
fill="#006400", name="CpG",
background.title="brown")
refGenes <- UcscTrack(genome=genome, chromosome=chr,
track="refGene", from=start, to=end,
trackType="GeneRegionTrack", rstarts="exonStarts",
rends="exonEnds", gene="name2", symbol="name2",
transcript="name", strand="strand", fill="#8282d2",
name="refSeq Genes", collapseTranscripts=TRUE,
showId=TRUE, background.title="brown")
ensembl <- useMart(biomart="ENSEMBL_MART_ENSEMBL",
host="dec2015.archive.ensembl.org")
ensembl <- useDataset("drerio_gene_ensembl",mart=ensembl)
biomTrack <- BiomartGeneRegionTrack(genome=genome, chromosome=chr,
biomart=ensembl,
start=start , end=end, name="Ensembl Genes")
data(axisTrack)
data(cpgIslands)
data(refGenes)
library(Gviz)
plotTracks(list(axisTrack, cpgIslands, refGenes),
collapseTranscripts=TRUE, shape="arrow",
transcriptAnnotation="symbol")
It is also possible to plot the annotation from an ordinary R object,
such as data.frame, GRanges, IRanges
or even from a local file.
Usually the gff file containing the gene annotation can be processed by
r Biocpkg("Gviz") directly.
For more details, please look into the vignette of r Biocpkg("Gviz").
CNEs horizon plot
dbName <- file.path(system.file("extdata", package="CNEr"),
"danRer10CNE.sqlite")
genome <- "danRer10"
windowSize <- 200L
minLength <- 50L
cneDanRer10Hg38_21_30 <-
CNEDensity(dbName=dbName,
tableName="danRer10_hg38_21_30",
whichAssembly="first", chr=chr, start=start,
end=end, windowSize=windowSize,
minLength=minLength)
cneDanRer10Hg38_45_50 <-
CNEDensity(dbName=dbName,
tableName="danRer10_hg38_45_50",
whichAssembly="first", chr=chr, start=start,
end=end, windowSize=windowSize,
minLength=minLength)
cneDanRer10Hg38_49_50 <-
CNEDensity(dbName=dbName,
tableName="danRer10_hg38_49_50",
whichAssembly="first", chr=chr, start=start,
end=end, windowSize=windowSize,
minLength=minLength)
cneDanRer10AstMex102_48_50 <-
CNEDensity(dbName=dbName,
tableName="AstMex102_danRer10_48_50",
whichAssembly="second", chr=chr, start=start,
end=end, windowSize=windowSize,
minLength=minLength)
cneDanRer10CteIde1_75_75 <-
CNEDensity(dbName=dbName,
tableName="cteIde1_danRer10_75_75",
whichAssembly="second", chr=chr, start=start,
end=end, windowSize=windowSize,
minLength=minLength)
dTrack1 <- DataTrack(range=cneDanRer10Hg38_21_30,
genome=genome, type="horiz",
horizon.scale=max(cneDanRer10Hg38_21_30$score)/3,
fill.horizon=c("#B41414", "#E03231", "#F7A99C",
"yellow", "orange", "red"),
name="human 21/30", background.title="brown")
dTrack2 <- DataTrack(range=cneDanRer10Hg38_45_50,
genome=genome, type="horiz",
horizon.scale=max(cneDanRer10Hg38_45_50$score)/2,
fill.horizon=c("#B41414", "#E03231", "#F7A99C",
"yellow", "orange", "red"),
name="human 45/50", background.title="brown")
dTrack3 <- DataTrack(range=cneDanRer10Hg38_49_50,
genome=genome, type="horiz",
horizon.scale=max(cneDanRer10Hg38_21_30$score)/3,
fill.horizon=c("#B41414", "#E03231", "#F7A99C",
"yellow", "orange", "red"),
name="human 49/50", background.title="brown")
dTrack4 <- DataTrack(range=cneDanRer10AstMex102_48_50,
genome=genome, type="horiz",
horizon.scale=max(cneDanRer10Hg38_21_30$score)/3,
fill.horizon=c("#B41414", "#E03231", "#F7A99C",
"yellow", "orange", "red"),
name="blind cave fish 48/50", background.title="brown")
dTrack5 <- DataTrack(range=cneDanRer10CteIde1_75_75,
genome=genome, type="horiz",
horizon.scale=max(cneDanRer10CteIde1_75_75$score)/3,
fill.horizon=c("#B41414", "#E03231", "#F7A99C",
"yellow", "orange", "red"),
name="grass carp 75/75", background.title="brown")
ht <- HighlightTrack(trackList=list(refGenes, dTrack5, dTrack4,
dTrack1, dTrack2, dTrack3),
start=c(24200000, 25200000, 26200000),
end=c(25100000, 26150000, 27000000),
chromosome =chr)
plotTracks(list(axisTrack, cpgIslands, ht),
collapseTranscripts=TRUE, shape="arrow",
transcriptAnnotation="symbol",
from=24000000, to=27000000)
From this horizon plot and when comparing Zebrafish with Human as a reference genome,
we notice that the genes barhl2, lmo4b and sox14 are surrounded by the density peaks of CNEs.
Conclusions
r Biocpkg("CNEr") efficiently identifies CNEs
and handles the corresponding objects conveniently in R.
Horizon plot shows a superior dynamic range to the standard density plots,
simultaneously revealing CNE clusters characterized
by vastly different levels of sequence conservation.
Such CNE density plots generated using precise locations of CNEs
can be used to identify genes involved in developmental regulation,
even for novel genes that are not yet annotated.
The following is the session info that generated this vignette:
sessionInfo()
References
Try the CNEr package in your browser
Any scripts or data that you put into this service are public.
CNEr documentation built on Nov. 8, 2020, 5:36 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
date = "`r doc_date()`" pkg = "`r pkg_ver('CNEr')`"
short=TRUE #if short==TRUE, do not echo code chunks debug=FALSE knitr::opts_chunk$set(echo=!short, warning=debug, message=debug, error=FALSE, cache.path = "cache/", fig.path = "figures/")
Introduction
Conserved noncoding elements (CNEs) are a pervasive class of elements clustering around genes with roles in development and differentiation in Metazoa [@Woolfe:2004ur]. While many have been shown to act as long-range developmental enhancers [@Sandelin:2004bd], the source of their extreme conservation remains unexplained. To study the evolutionary dynamics of these elements and their relationship to the genes around which they cluster, it is essential to be able to produce genome-wide sets of these elements for a large number of species comparisons, each with multiple size and conservation thresholds.
This r Biocpkg("CNEr") package aims to detect CNEs and
visualise them along the genome.
For performance reasons, the implementation of CNEs detection
and corresponding I/O functions are primarily written as C extensions to R.
We have used r Biocpkg("CNEr") to produce sets of CNEs by scanning pairwise whole-genome net alignments
with multiple reference species,
each with two different window sizes and a range of minimum identity thresholds.
Then, to pinpoint the boundaries of CNE arrays,
we compute the CNE densities as the percentages of length
covered by CNEs within a user specified window size.
Finally, we describe a novel visualisation method using horizon plot tracks
that shows a superior dynamic range to the standard density plots,
simultaneously revealing CNE clusters characterized
by vastly different levels of sequence conservation.
Such CNE density plots generated using precise locations of CNEs
can be used to identify genes involved in developmental regulation,
even novel genes that are not annotated yet.
Workflow of the package
This section briefly demonstrates the pipeline of CNE identification and visualisation. More detailed usage of each step is described in following sections with a concise example of CNE identification and visualisation for the "barhl2" and "sox14" (chr6:24,000,000..27,000,000) loci in Zebrafish (danRer10) genome against Human (hg38).
CNE identification
- Preparation of axtNets: the axtNet files can be downloaded from UCSC or generated by this package.
- Identification of conserved noncoding regions: scan the axt alignments for the regions, with minimal I identities over C columns, which do not overlap with annotated exons/repeats.
- Merging overlapped elements : scanned elements from two pairs of alignments (one from each genome as reference) that overlap on both genomes are merged.
- Removal of unannotated repetitive sequences: realign the elements back to respective genome with BLAT and discard the elements which exceed a certain number of hits on either genome.
CNE visualisation
- Display parameters: chromosome, start, end, smooth window size.
- Horizon plot: visualise CNE densities with the package
r Biocpkg("Gviz").
Input
The minimal input for r Biocpkg("CNEr") includes the whole genome pairwise alignment of two assemblies, axt
net files.
UCSC already provides a set of precomputed axt files on http://hgdownload.soe.ucsc.edu/downloads.html for most of popular genomes.
In case the axt net files are not available from UCSC, you can always generate the axt net files by following another vignette "Pairwise whole genome alignment" in this r Biocpkg("CNEr") package.
Another essential information is the annotation of exons and repeats, which could be retrieved from various resources.
axt alignment file
During the development of this package, there was no suitable class to store the axt alignments in Bioconductor.
Hence, we created two new S4 classes, Axt and GRangePairs, to easily manipulate the axt alignment files.
GRangePairs is designed to hold a pair of GRanges objects, which have the same length.
It builds on the Pairs class of Bioconductor and inherits many useful methods from it.
This Axt class inherits from GRangePairs with no extra slots to hold the content from axt files, but many specific methods are created for Axt.
The ranges of the target and the query organism are stored in two GRanges objects with alignment sequences as metadata columns.
The Blastz scores and the widths of the alignments are stored in metadata columns of GRangePairs.
For more information on the usage of these two classes, please refer to the documentation.
To read axt file into R,
r Biocpkg("CNEr") provides readAxt function for highly efficient reading.
This function is built on a backend C code of Kent's utilities [@Kent:2002bw].
The axt alignment files can be either gzippped or in plain text file.
The alignments between two genomes can also be in one big file or in several files, such as "chr1.hg19.mm10.net.axt.gz", "chr2.hg19.mm10.net.axt.gz", etc.
library(CNEr) ## These axt files are specially prepared for the region ## (chr6:24,000,000..27,000,000) axtFilesHg38DanRer10 <- file.path(system.file("extdata", package="CNEr"), "hg38.danRer10.net.axt") axtFilesDanRer10Hg38 <- file.path(system.file("extdata", package="CNEr"), "danRer10.hg38.net.axt")
axtHg38DanRer10 <- readAxt(axtFilesHg38DanRer10) axtDanRer10Hg38 <- readAxt(axtFilesDanRer10Hg38)
## Axt class is shown in UCSC axt format
axtHg38DanRer10
axtDanRer10Hg38
## Distribution of matched alignments; Given an Axt alignment, plot a heatmap with percentage of each matched alignment matchDistribution(axtHg38DanRer10) matchDistribution(axtDanRer10Hg38)
## Example of chr4 on hg19 and galGal3 ## The synteny of human and zebrafish is not quite obvious on the dotplot. library(BSgenome.Hsapiens.UCSC.hg19) library(BSgenome.Ggallus.UCSC.galGal3) fn <- file.path(system.file("extdata", package="CNEr"), "chr4.hg19.galGal3.net.axt.gz") axt <- readAxt(fn, tAssemblyFn=file.path(system.file("extdata", package="BSgenome.Hsapiens.UCSC.hg19"), "single_sequences.2bit"), qAssemblyFn=file.path(system.file("extdata", package="BSgenome.Ggallus.UCSC.galGal3"), "single_sequences.2bit")) library(GenomeInfoDb) syntenicDotplot(axt, firstChrs=c("chr4"), secondChrs="chr4", type="dot")
There are methods defined for handling Axt objects, including subsetting, output to axt files.
More details can be found in the man page.
Filtering information
The gene annotation information, including exons and repeats, is used to filter out the undesired regions. Here we summarise a table of filtering information we used:
Assembly | Name | Exon | Repeat
----------- | ----------- | ----------------------------------------- | ---------
hg38 | Human | RefSeq Genes, Ensembl Genes, UCSC Known Genes | RepeatMasker
mm10 | Mouse | RefSeq Genes, Ensembl Genes, UCSC Known Genes | RepeatMasker
xenTro3 | Frog | RefSeq Genes, Ensembl Genes | RepeatMasker
tetNig2 | Tetraodon | Ensembl Genes | |
canFam3 | Dog | RefSeq Genes, Ensembl Genes | RepeatMasker
galGal4 | Chicken | RefSeq Genes, Ensembl Genes | RepeatMasker
danRer10 | Zebrafish | RefSeq Genes, Ensembl Genes | RepeatMasker
fr3 | Fugu | RefSeq Genes | RepeatMasker
anoCar2 | Lizard | Ensembl Genes | RepeatMasker
equCab2 | Horse | RefSeq Genes, Ensembl Genes | RepeatMasker
oryLat2 | Medaka | RefSeq Genes, Ensembl Genes | RepeatMasker
monDom5 | Opossum | RefSeq Genes, Ensembl Genes | RepeatMasker
gasAcu1 | Stickleback | RefSeq Genes, Ensembl Genes | RepeatMasker
rn5 | Rat | RefSeq Genes, Ensembl Genes | RepeatMasker
dm3 | D. melanogaster | RefSeq Genes, Ensembl Genes | RepeatMasker
droAna2 | D. ananassae | | RepeatMasker
dp3 | D. pseudoobscura | | RepeatMasker
ce4 | C. elegans | RefSeq Genes | RepeatMasker
cb3 | C. briggsae | | RepeatMasker
caeRem2 | C. remanei | | RepeatMasker
caePb1 | C. brenneri | | RepeatMasker
For the sake of simplicity, all the information listed above can be fetched easily
with Bioconductor package r Biocpkg("rtracklayer"), r Biocpkg("biomaRt") and
precompiled Bioconductor annotation packages.
A few examples are given here:
## To fetch rmsk table from UCSC library(rtracklayer) mySession <- browserSession("UCSC") genome(mySession) <- "hg38" hg38.rmsk <- getTable(ucscTableQuery(mySession, track="RepeatMasker", table="rmsk")) hg38.rmskGRanges <- GRanges(seqnames=hg38.rmsk$genoName, ## The UCSC coordinate is 0-based. ranges=IRanges(start=hg38.rmsk$genoStart+1, end=hg38.rmsk$genoEnd), strand=hg38.rmsk$strand) ## To fetch ensembl gene exons from BioMart library(biomaRt) ensembl <- useMart(biomart="ENSEMBL_MART_ENSEMBL", host="dec2015.archive.ensembl.org") ensembl <- useDataset("hsapiens_gene_ensembl",mart=ensembl) attributes <- listAttributes(ensembl) exons <- getBM(attributes=c("chromosome_name", "exon_chrom_start", "exon_chrom_end", "strand"), mart=ensembl) exonsRanges <- GRanges(seqnames=exons$chromosome_name, ranges=IRanges(start=exons$exon_chrom_start, end=exons$exon_chrom_end), strand=ifelse(exons$strand==1L, "+", "-") ) seqlevelsStyle(exonsRanges) <- "UCSC" ## Use the existing Bioconductor annotation package for hg38 library(TxDb.Hsapiens.UCSC.hg38.knownGene) exonsRanges <- exons(TxDb.Hsapiens.UCSC.hg38.knownGene)
The regions to filter out can also be provided in a bed file.
To import the bed file into r Biocpkg("GRanges") in R,
r Biocpkg("rtracklayer") provides a general function
import.bed to do that.
Since only the chromosome names, start and end coordinates are used in r Biocpkg("CNEr"),
we provide a more efficient readBed function.
## Existing bed file for chr6:24,000,000..27,000,000 of Zebrafish danRer10 bedDanRer10Fn <- file.path(system.file("extdata", package="CNEr"), "filter_regions.danRer10.bed") danRer10Filter <- readBed(bedDanRer10Fn) danRer10Filter ## Existing bed file for alignment region in Human hg38 against ## chr6:24,000,000..27,000,000 of danRer10 bedHg38Fn <- file.path(system.file("extdata", package="CNEr"), "filter_regions.hg38.bed") hg38Filter <- readBed(bedHg38Fn) hg38Filter
Creating a CNE class
We designed a CNE class to store all metadata of running the pipeline for identifying a set of CNEs between two species, including the intermediate and final results.
CNE can be created by providing the paths of the twoBit files of two assemblies, and
the paths of axt files, with each assembly as reference.
## Here we have the twoBit files from Bioconductor package ## BSgenome.Drerio.UCSC.danRer10 and BSgenome.Hsapiens.UCSC.hg38 cneDanRer10Hg38 <- CNE( assembly1Fn=file.path(system.file("extdata", package="BSgenome.Drerio.UCSC.danRer10"), "single_sequences.2bit"), assembly2Fn=file.path(system.file("extdata", package="BSgenome.Hsapiens.UCSC.hg38"), "single_sequences.2bit"), axt12Fn=axtFilesDanRer10Hg38, axt21Fn=axtFilesHg38DanRer10, cutoffs1=8L, cutoffs2=4L) cneDanRer10Hg38
Note: the order of assemblies when creating CNE object is important.
Here we have danRer10 as assembly1 and hg38 as assembly2.
Then the axt12Fn contains the axt alignment with assembly1 danRer10 as reference and axt21Fn contains the alignment with assembly2 hg38 as reference.
The cutoffs1 and cutoffs2 are the maximal number of hits during the realignment in later steps.
Because zebrafish has undergone additional whole genome duplication compared to human, the cutoffs of zebrafish also doubles the cutoffs of human.
CNE identification
In this section, we will go through the details of CNE identification.
Scan axt alignments
Detecting CNEs highly relies on the whole-genome pairwise net alignments. To correct the bias of a chosen genome (which bias?) and capture the duplicated CNEs during genome evolution, we scan two sets of nets for each pairwise comparison, one as reference from each of the genomes.
We identify CNEs by scanning the alignments for regions with at least I identities over C alignment columns. Because different genes and loci may favor various similarity scores, we usually scan at two diffrent window sizes 30 and 50 with several similarity criterias (I/C) range from 70% to 100%.
identities <- c(45L, 48L, 49L) windows <- c(50L, 50L, 50L) ## Here danRer10Filter is tFilter since danRer10 is assembly1 cneListDanRer10Hg38 <- ceScan(x=cneDanRer10Hg38, tFilter=danRer10Filter, qFilter=hg38Filter, window=windows, identity=identities)
At this stage, a list of CNE is returned from ceScan, which contains the preliminary two sets of CNEs from a pair of axt alignments.
We can examine the intermediate CNEs by
## CNEs from the alignments with danRer10 as reference CNE12(cneListDanRer10Hg38[["45_50"]]) ## CNEs from the alignments with hg38 as reference CNE21(cneListDanRer10Hg38[["45_50"]])
In the result table, even though the strand for query element can be negative, the coordinate for that query element is already on the positive strand.
It is essential to scan two sets of pairwise net alignments with each assembly as reference, in order not to miss any duplicated elements in either lineage. This is particularly important for the comparison between teleost fishes and other vertebrates, because one (for instance, the case of zebrafish) or two (the case of common carp) extra whole genome duplications occured.
Merge CNEs
As we perform two rounds of CNE detection with each genome as reference, some conserved elements overlap on both genomes and should be removed. Elements, however, that overlap only on one of the genomes are kept, so that duplicated elements remain distinct.
cneMergedListDanRer10Hg38 <- lapply(cneListDanRer10Hg38, cneMerge)
Realignment of CNEs
Some CNEs might be unannotated repeats. To remove them, currently we use blat [@Kent:2002jd] to realign each sequence of CNEs against the respective genomes. When the number of matches exceeds a certain threshold, for instance 8, that CNE will be discarded.
This step can be very time-consuming when the number of CNEs is large. Other alignment methods could also be considered, for example Bowtie2 or BWA (provided that they are installed on the user's machine)
cneFinalListDanRer10Hg38 <- lapply(cneMergedListDanRer10Hg38, blatCNE)
CNE storage and query
As the computation of CNEs from the whole pipeline and the preparation of annotation package can be very time-consuming, for a smoother visualisation experience, we decided to use a local SQLite database in order to store CNEs.
Since the CNEs data.frame is a table, it can be imported into a SQL table.
To speed up the query from the SQL database,
the bin indexing system is adopted.
For more information, please refer to the paper [@Kent:2002bw]
and genomewiki.
## on individual tables dbName <- tempfile() data(cneFinalListDanRer10Hg38) tableNames <- paste("danRer10", "hg38", names(cneFinalListDanRer10Hg38), sep="_") for(i in 1:length(cneFinalListDanRer10Hg38)){ saveCNEToSQLite(cneFinalListDanRer10Hg38[[i]], dbName, tableNames[i], overwrite=TRUE) }
When querying results from the local SQLite database based on the chr,
coordinates and other criterias,
a GRanges object is returned.
chr <- "chr6" start <- 24000000L end <- 27000000L minLength <- 50L tableName <- "danRer10_hg38_45_50" fetchedCNERanges <- readCNERangesFromSQLite(dbName, tableName, chr, start, end, whichAssembly="first", minLength=minLength) fetchedCNERanges
CNE length distribution
As the lengths of CNEs [@Salerno:2006sc] and the distances between consecutive elements [@Polychronopoulos:2014pl] exhibit power-law distributions, we implemented a function that might be useful in showing interesting patterns in the distribution of the identified elements.
dbName <- file.path(system.file("extdata", package="CNEr"), "danRer10CNE.sqlite") tAssemblyFn <- file.path(system.file("extdata", package="BSgenome.Drerio.UCSC.danRer10"), "single_sequences.2bit") qAssemblyFn <- file.path(system.file("extdata", package="BSgenome.Hsapiens.UCSC.hg38"), "single_sequences.2bit") cneGRangePairs <- readCNERangesFromSQLite(dbName=dbName, tableName="danRer10_hg38_45_50", tAssemblyFn=tAssemblyFn, qAssemblyFn=qAssemblyFn) plotCNEWidth(cneGRangePairs)
Genomic distribution of CNEs along the chromosome
CNEs tend to form clusters. A quick check of the genomic distribution of CNEs is available.
plotCNEDistribution(first(cneGRangePairs))
Output of bed and bedGraph files
For visualisation in other Genome Browser, we provide functions to generate the CNE in bed files and CNE density in bedGraph files. For example, to get the first 1000 coordinates of CNEs:
makeCNEDensity(cneGRangePairs[1:1000])
CNEs visualisation
To visualise CNEs alongside other gene annotations,
we choose to use the Bioconductor package r Biocpkg("Gviz") in this vignette.
r Biocpkg("Gviz"), based on the r CRANpkg("grid") graphics scheme,
is a very powerful package for plotting data and annotation information
along genomic coordinates.
The functionality of integrating publicly available genome annotation data,
such as UCSC or Ensembl,
significantly reduced the burden of preparing annotations for common assemblies.
Since the Bioconductor release 2.13 of r Biocpkg("Gviz"),
it provides the data track in horizon plot,
which exactly meets our needs for visualisation of CNEs density plots.
For more detailed usage, please check the vignette or manual of r Biocpkg("Gviz").
Another option for visualisation is the package r Biocpkg("ggbio"),
which is based on r CRANpkg("ggplot2").
The advantage of r Biocpkg("ggbio") is the simplicity of
adding any customised r CRANpkg("ggplot2") style track
into the plot without tuning the coordinate systems.
The densities generated in the following section can be easily plot in the
horizon plot.
A short straightforward tutorial regarding horizon plot
in ggplot2 format
is available from
http://timelyportfolio.blogspot.co.uk/2012/08/horizon-on-ggplot2.html.
Gene annotation visualisation
For the example case of hg38 vs danRer10 in this vignette, we choose danRer10 as the reference and show the range of developmental gene barhl2 and sox14.
library(Gviz) library(biomaRt) genome <- "danRer10" axisTrack <- GenomeAxisTrack() cpgIslands <- UcscTrack(genome=genome, chromosome=chr, track="cpgIslandExt", from=start, to=end, trackType="AnnotationTrack", start="chromStart", end="chromEnd", id="name", shape="box", showId=FALSE, fill="#006400", name="CpG", background.title="brown") refGenes <- UcscTrack(genome=genome, chromosome=chr, track="refGene", from=start, to=end, trackType="GeneRegionTrack", rstarts="exonStarts", rends="exonEnds", gene="name2", symbol="name2", transcript="name", strand="strand", fill="#8282d2", name="refSeq Genes", collapseTranscripts=TRUE, showId=TRUE, background.title="brown") ensembl <- useMart(biomart="ENSEMBL_MART_ENSEMBL", host="dec2015.archive.ensembl.org") ensembl <- useDataset("drerio_gene_ensembl",mart=ensembl) biomTrack <- BiomartGeneRegionTrack(genome=genome, chromosome=chr, biomart=ensembl, start=start , end=end, name="Ensembl Genes")
data(axisTrack) data(cpgIslands) data(refGenes)
library(Gviz) plotTracks(list(axisTrack, cpgIslands, refGenes), collapseTranscripts=TRUE, shape="arrow", transcriptAnnotation="symbol")
It is also possible to plot the annotation from an ordinary R object,
such as data.frame, GRanges, IRanges
or even from a local file.
Usually the gff file containing the gene annotation can be processed by
r Biocpkg("Gviz") directly.
For more details, please look into the vignette of r Biocpkg("Gviz").
CNEs horizon plot
dbName <- file.path(system.file("extdata", package="CNEr"), "danRer10CNE.sqlite") genome <- "danRer10" windowSize <- 200L minLength <- 50L cneDanRer10Hg38_21_30 <- CNEDensity(dbName=dbName, tableName="danRer10_hg38_21_30", whichAssembly="first", chr=chr, start=start, end=end, windowSize=windowSize, minLength=minLength) cneDanRer10Hg38_45_50 <- CNEDensity(dbName=dbName, tableName="danRer10_hg38_45_50", whichAssembly="first", chr=chr, start=start, end=end, windowSize=windowSize, minLength=minLength) cneDanRer10Hg38_49_50 <- CNEDensity(dbName=dbName, tableName="danRer10_hg38_49_50", whichAssembly="first", chr=chr, start=start, end=end, windowSize=windowSize, minLength=minLength) cneDanRer10AstMex102_48_50 <- CNEDensity(dbName=dbName, tableName="AstMex102_danRer10_48_50", whichAssembly="second", chr=chr, start=start, end=end, windowSize=windowSize, minLength=minLength) cneDanRer10CteIde1_75_75 <- CNEDensity(dbName=dbName, tableName="cteIde1_danRer10_75_75", whichAssembly="second", chr=chr, start=start, end=end, windowSize=windowSize, minLength=minLength)
dTrack1 <- DataTrack(range=cneDanRer10Hg38_21_30, genome=genome, type="horiz", horizon.scale=max(cneDanRer10Hg38_21_30$score)/3, fill.horizon=c("#B41414", "#E03231", "#F7A99C", "yellow", "orange", "red"), name="human 21/30", background.title="brown") dTrack2 <- DataTrack(range=cneDanRer10Hg38_45_50, genome=genome, type="horiz", horizon.scale=max(cneDanRer10Hg38_45_50$score)/2, fill.horizon=c("#B41414", "#E03231", "#F7A99C", "yellow", "orange", "red"), name="human 45/50", background.title="brown") dTrack3 <- DataTrack(range=cneDanRer10Hg38_49_50, genome=genome, type="horiz", horizon.scale=max(cneDanRer10Hg38_21_30$score)/3, fill.horizon=c("#B41414", "#E03231", "#F7A99C", "yellow", "orange", "red"), name="human 49/50", background.title="brown") dTrack4 <- DataTrack(range=cneDanRer10AstMex102_48_50, genome=genome, type="horiz", horizon.scale=max(cneDanRer10Hg38_21_30$score)/3, fill.horizon=c("#B41414", "#E03231", "#F7A99C", "yellow", "orange", "red"), name="blind cave fish 48/50", background.title="brown") dTrack5 <- DataTrack(range=cneDanRer10CteIde1_75_75, genome=genome, type="horiz", horizon.scale=max(cneDanRer10CteIde1_75_75$score)/3, fill.horizon=c("#B41414", "#E03231", "#F7A99C", "yellow", "orange", "red"), name="grass carp 75/75", background.title="brown")
ht <- HighlightTrack(trackList=list(refGenes, dTrack5, dTrack4, dTrack1, dTrack2, dTrack3), start=c(24200000, 25200000, 26200000), end=c(25100000, 26150000, 27000000), chromosome =chr) plotTracks(list(axisTrack, cpgIslands, ht), collapseTranscripts=TRUE, shape="arrow", transcriptAnnotation="symbol", from=24000000, to=27000000)
From this horizon plot and when comparing Zebrafish with Human as a reference genome, we notice that the genes barhl2, lmo4b and sox14 are surrounded by the density peaks of CNEs.
Conclusions
r Biocpkg("CNEr") efficiently identifies CNEs
and handles the corresponding objects conveniently in R.
Horizon plot shows a superior dynamic range to the standard density plots,
simultaneously revealing CNE clusters characterized
by vastly different levels of sequence conservation.
Such CNE density plots generated using precise locations of CNEs
can be used to identify genes involved in developmental regulation,
even for novel genes that are not yet annotated.
The following is the session info that generated this vignette:
sessionInfo()
References
Try the CNEr package in your browser
Any scripts or data that you put into this service are public.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.