CNE identification and visualisation

date = "`r doc_date()`"
pkg = "`r pkg_ver('CNEr')`"
short=TRUE #if short==TRUE, do not echo code chunks
knitr::opts_chunk$set(echo=!short, warning=debug, message=debug, error=FALSE,
               cache.path = "cache/", 
               fig.path = "figures/")


Conserved noncoding elements (CNEs) are a pervasive class of elements clustering around genes with roles in development and differentiation in Metazoa [@Woolfe:2004ur]. While many have been shown to act as long-range developmental enhancers [@Sandelin:2004bd], the source of their extreme conservation remains unexplained. To study the evolutionary dynamics of these elements and their relationship to the genes around which they cluster, it is essential to be able to produce genome-wide sets of these elements for a large number of species comparisons, each with multiple size and conservation thresholds.

This r Biocpkg("CNEr") package aims to detect CNEs and visualise them along the genome. For performance reasons, the implementation of CNEs detection and corresponding I/O functions are primarily written as C extensions to R. We have used r Biocpkg("CNEr") to produce sets of CNEs by scanning pairwise whole-genome net alignments with multiple reference species, each with two different window sizes and a range of minimum identity thresholds. Then, to pinpoint the boundaries of CNE arrays, we compute the CNE densities as the percentages of length covered by CNEs within a user specified window size. Finally, we describe a novel visualisation method using horizon plot tracks that shows a superior dynamic range to the standard density plots, simultaneously revealing CNE clusters characterized by vastly different levels of sequence conservation. Such CNE density plots generated using precise locations of CNEs can be used to identify genes involved in developmental regulation, even novel genes that are not annotated yet.

Workflow of the package

This section briefly demonstrates the pipeline of CNE identification and visualisation. More detailed usage of each step is described in following sections with a concise example of CNE identification and visualisation for the "barhl2" and "sox14" (chr6:24,000,000..27,000,000) loci in Zebrafish (danRer10) genome against Human (hg38).

CNE identification

  1. Preparation of axtNets: the axtNet files can be downloaded from UCSC or generated by this package.
  2. Identification of conserved noncoding regions: scan the axt alignments for the regions, with minimal I identities over C columns, which do not overlap with annotated exons/repeats.
  3. Merging overlapped elements : scanned elements from two pairs of alignments (one from each genome as reference) that overlap on both genomes are merged.
  4. Removal of unannotated repetitive sequences: realign the elements back to respective genome with BLAT and discard the elements which exceed a certain number of hits on either genome.

CNE visualisation

  1. Display parameters: chromosome, start, end, smooth window size.
  2. Horizon plot: visualise CNE densities with the package r Biocpkg("Gviz").


The minimal input for r Biocpkg("CNEr") includes the whole genome pairwise alignment of two assemblies, axt net files. UCSC already provides a set of precomputed axt files on for most of popular genomes. In case the axt net files are not available from UCSC, you can always generate the axt net files by following another vignette "Pairwise whole genome alignment" in this r Biocpkg("CNEr") package.

Another essential information is the annotation of exons and repeats, which could be retrieved from various resources.

axt alignment file

During the development of this package, there was no suitable class to store the axt alignments in Bioconductor. Hence, we created two new S4 classes, Axt and GRangePairs, to easily manipulate the axt alignment files.

GRangePairs is designed to hold a pair of GRanges objects, which have the same length. It builds on the Pairs class of Bioconductor and inherits many useful methods from it. This Axt class inherits from GRangePairs with no extra slots to hold the content from axt files, but many specific methods are created for Axt. The ranges of the target and the query organism are stored in two GRanges objects with alignment sequences as metadata columns. The Blastz scores and the widths of the alignments are stored in metadata columns of GRangePairs. For more information on the usage of these two classes, please refer to the documentation.

To read axt file into R, r Biocpkg("CNEr") provides readAxt function for highly efficient reading. This function is built on a backend C code of Kent's utilities [@Kent:2002bw]. The axt alignment files can be either gzippped or in plain text file. The alignments between two genomes can also be in one big file or in several files, such as "", "", etc.

## These axt files are specially prepared for the region
## (chr6:24,000,000..27,000,000)
axtFilesHg38DanRer10 <- file.path(system.file("extdata", package="CNEr"), 
axtFilesDanRer10Hg38 <- file.path(system.file("extdata", package="CNEr"), 
axtHg38DanRer10 <- readAxt(axtFilesHg38DanRer10)
axtDanRer10Hg38 <- readAxt(axtFilesDanRer10Hg38)
## Axt class is shown in UCSC axt format
## Distribution of matched alignments; Given an Axt alignment, plot a heatmap with percentage of each matched alignment
## Example of chr4 on hg19 and galGal3
## The synteny of human and zebrafish is not quite obvious on the dotplot.
fn <- file.path(system.file("extdata", package="CNEr"),
axt <- readAxt(fn, 
syntenicDotplot(axt, firstChrs=c("chr4"), secondChrs="chr4", type="dot")

There are methods defined for handling Axt objects, including subsetting, output to axt files. More details can be found in the man page.

Filtering information

The gene annotation information, including exons and repeats, is used to filter out the undesired regions. Here we summarise a table of filtering information we used:

Assembly | Name | Exon | Repeat ----------- | ----------- | ----------------------------------------- | --------- hg38 | Human | RefSeq Genes, Ensembl Genes, UCSC Known Genes | RepeatMasker mm10 | Mouse | RefSeq Genes, Ensembl Genes, UCSC Known Genes | RepeatMasker
xenTro3 | Frog | RefSeq Genes, Ensembl Genes | RepeatMasker tetNig2 | Tetraodon | Ensembl Genes | | canFam3 | Dog | RefSeq Genes, Ensembl Genes | RepeatMasker galGal4 | Chicken | RefSeq Genes, Ensembl Genes | RepeatMasker danRer10 | Zebrafish | RefSeq Genes, Ensembl Genes | RepeatMasker fr3 | Fugu | RefSeq Genes | RepeatMasker anoCar2 | Lizard | Ensembl Genes | RepeatMasker equCab2 | Horse | RefSeq Genes, Ensembl Genes | RepeatMasker oryLat2 | Medaka | RefSeq Genes, Ensembl Genes | RepeatMasker monDom5 | Opossum | RefSeq Genes, Ensembl Genes | RepeatMasker gasAcu1 | Stickleback | RefSeq Genes, Ensembl Genes | RepeatMasker rn5 | Rat | RefSeq Genes, Ensembl Genes | RepeatMasker dm3 | D. melanogaster | RefSeq Genes, Ensembl Genes | RepeatMasker droAna2 | D. ananassae | | RepeatMasker dp3 | D. pseudoobscura | | RepeatMasker ce4 | C. elegans | RefSeq Genes | RepeatMasker cb3 | C. briggsae | | RepeatMasker caeRem2 | C. remanei | | RepeatMasker
caePb1 | C. brenneri | | RepeatMasker

For the sake of simplicity, all the information listed above can be fetched easily with Bioconductor package r Biocpkg("rtracklayer"), r Biocpkg("biomaRt") and precompiled Bioconductor annotation packages. A few examples are given here:

## To fetch rmsk table from UCSC
mySession <- browserSession("UCSC")
genome(mySession) <- "hg38"
hg38.rmsk <- getTable(ucscTableQuery(mySession, track="RepeatMasker",
hg38.rmskGRanges <- GRanges(seqnames=hg38.rmsk$genoName,
                            ## The UCSC coordinate is 0-based.
## To fetch ensembl gene exons from BioMart
ensembl <- useMart(biomart="ENSEMBL_MART_ENSEMBL",
ensembl <-  useDataset("hsapiens_gene_ensembl",mart=ensembl)
attributes <- listAttributes(ensembl)
exons <- getBM(attributes=c("chromosome_name", "exon_chrom_start",
                            "exon_chrom_end", "strand"), 
exonsRanges <- GRanges(seqnames=exons$chromosome_name,
                       strand=ifelse(exons$strand==1L, "+", "-")
seqlevelsStyle(exonsRanges) <- "UCSC"
## Use the existing Bioconductor annotation package for hg38
exonsRanges <- exons(TxDb.Hsapiens.UCSC.hg38.knownGene)

The regions to filter out can also be provided in a bed file. To import the bed file into r Biocpkg("GRanges") in R, r Biocpkg("rtracklayer") provides a general function import.bed to do that. Since only the chromosome names, start and end coordinates are used in r Biocpkg("CNEr"), we provide a more efficient readBed function.

## Existing bed file for chr6:24,000,000..27,000,000 of Zebrafish danRer10
bedDanRer10Fn <- file.path(system.file("extdata", package="CNEr"), 
danRer10Filter <- readBed(bedDanRer10Fn)

## Existing bed file for alignment region in Human hg38 against
## chr6:24,000,000..27,000,000 of danRer10
bedHg38Fn <- file.path(system.file("extdata", package="CNEr"), 
hg38Filter <- readBed(bedHg38Fn)

Creating a CNE class

We designed a CNE class to store all metadata of running the pipeline for identifying a set of CNEs between two species, including the intermediate and final results. CNE can be created by providing the paths of the twoBit files of two assemblies, and the paths of axt files, with each assembly as reference.

## Here we have the twoBit files from Bioconductor package
## BSgenome.Drerio.UCSC.danRer10 and BSgenome.Hsapiens.UCSC.hg38
cneDanRer10Hg38 <- CNE(
  axt12Fn=axtFilesDanRer10Hg38, axt21Fn=axtFilesHg38DanRer10,
  cutoffs1=8L, cutoffs2=4L)

Note: the order of assemblies when creating CNE object is important. Here we have danRer10 as assembly1 and hg38 as assembly2. Then the axt12Fn contains the axt alignment with assembly1 danRer10 as reference and axt21Fn contains the alignment with assembly2 hg38 as reference. The cutoffs1 and cutoffs2 are the maximal number of hits during the realignment in later steps. Because zebrafish has undergone additional whole genome duplication compared to human, the cutoffs of zebrafish also doubles the cutoffs of human.

CNE identification

In this section, we will go through the details of CNE identification.

Scan axt alignments

Detecting CNEs highly relies on the whole-genome pairwise net alignments. To correct the bias of a chosen genome (which bias?) and capture the duplicated CNEs during genome evolution, we scan two sets of nets for each pairwise comparison, one as reference from each of the genomes.

We identify CNEs by scanning the alignments for regions with at least I identities over C alignment columns. Because different genes and loci may favor various similarity scores, we usually scan at two diffrent window sizes 30 and 50 with several similarity criterias (I/C) range from 70% to 100%.

identities <- c(45L, 48L, 49L)
windows <- c(50L, 50L, 50L)
## Here danRer10Filter is tFilter since danRer10 is assembly1
cneListDanRer10Hg38 <- ceScan(x=cneDanRer10Hg38, tFilter=danRer10Filter,
                              window=windows, identity=identities)

At this stage, a list of CNE is returned from ceScan, which contains the preliminary two sets of CNEs from a pair of axt alignments. We can examine the intermediate CNEs by

## CNEs from the alignments with danRer10 as reference
## CNEs from the alignments with hg38 as reference

In the result table, even though the strand for query element can be negative, the coordinate for that query element is already on the positive strand.

It is essential to scan two sets of pairwise net alignments with each assembly as reference, in order not to miss any duplicated elements in either lineage. This is particularly important for the comparison between teleost fishes and other vertebrates, because one (for instance, the case of zebrafish) or two (the case of common carp) extra whole genome duplications occured.

Merge CNEs

As we perform two rounds of CNE detection with each genome as reference, some conserved elements overlap on both genomes and should be removed. Elements, however, that overlap only on one of the genomes are kept, so that duplicated elements remain distinct.

cneMergedListDanRer10Hg38 <- lapply(cneListDanRer10Hg38, cneMerge)

Realignment of CNEs

Some CNEs might be unannotated repeats. To remove them, currently we use blat [@Kent:2002jd] to realign each sequence of CNEs against the respective genomes. When the number of matches exceeds a certain threshold, for instance 8, that CNE will be discarded.

This step can be very time-consuming when the number of CNEs is large. Other alignment methods could also be considered, for example Bowtie2 or BWA (provided that they are installed on the user's machine)

cneFinalListDanRer10Hg38 <- lapply(cneMergedListDanRer10Hg38, blatCNE)

CNE storage and query

As the computation of CNEs from the whole pipeline and the preparation of annotation package can be very time-consuming, for a smoother visualisation experience, we decided to use a local SQLite database in order to store CNEs.

Since the CNEs data.frame is a table, it can be imported into a SQL table. To speed up the query from the SQL database, the bin indexing system is adopted. For more information, please refer to the paper [@Kent:2002bw] and genomewiki.

## on individual tables
dbName <- tempfile()
tableNames <- paste("danRer10", "hg38", names(cneFinalListDanRer10Hg38),
for(i in 1:length(cneFinalListDanRer10Hg38)){
    saveCNEToSQLite(cneFinalListDanRer10Hg38[[i]], dbName, tableNames[i],

When querying results from the local SQLite database based on the chr, coordinates and other criterias, a GRanges object is returned.

chr <- "chr6"
start <- 24000000L
end <-  27000000L
minLength <- 50L
tableName <- "danRer10_hg38_45_50"
fetchedCNERanges <- readCNERangesFromSQLite(dbName, tableName, chr, 
                                            start, end, whichAssembly="first",

CNE length distribution

As the lengths of CNEs [@Salerno:2006sc] and the distances between consecutive elements [@Polychronopoulos:2014pl] exhibit power-law distributions, we implemented a function that might be useful in showing interesting patterns in the distribution of the identified elements.

dbName <- file.path(system.file("extdata", package="CNEr"),
tAssemblyFn <- file.path(system.file("extdata",
qAssemblyFn <- file.path(system.file("extdata",
cneGRangePairs <- readCNERangesFromSQLite(dbName=dbName, 

Genomic distribution of CNEs along the chromosome

CNEs tend to form clusters. A quick check of the genomic distribution of CNEs is available.


Output of bed and bedGraph files

For visualisation in other Genome Browser, we provide functions to generate the CNE in bed files and CNE density in bedGraph files. For example, to get the first 1000 coordinates of CNEs:


CNEs visualisation

To visualise CNEs alongside other gene annotations, we choose to use the Bioconductor package r Biocpkg("Gviz") in this vignette. r Biocpkg("Gviz"), based on the r CRANpkg("grid") graphics scheme, is a very powerful package for plotting data and annotation information along genomic coordinates. The functionality of integrating publicly available genome annotation data, such as UCSC or Ensembl, significantly reduced the burden of preparing annotations for common assemblies. Since the Bioconductor release 2.13 of r Biocpkg("Gviz"), it provides the data track in horizon plot, which exactly meets our needs for visualisation of CNEs density plots. For more detailed usage, please check the vignette or manual of r Biocpkg("Gviz").

Another option for visualisation is the package r Biocpkg("ggbio"), which is based on r CRANpkg("ggplot2"). The advantage of r Biocpkg("ggbio") is the simplicity of adding any customised r CRANpkg("ggplot2") style track into the plot without tuning the coordinate systems. The densities generated in the following section can be easily plot in the horizon plot. A short straightforward tutorial regarding horizon plot in ggplot2 format is available from

Gene annotation visualisation

For the example case of hg38 vs danRer10 in this vignette, we choose danRer10 as the reference and show the range of developmental gene barhl2 and sox14.

genome <- "danRer10"
axisTrack <- GenomeAxisTrack()
cpgIslands <- UcscTrack(genome=genome, chromosome=chr,
                        track="cpgIslandExt", from=start, to=end,
                        trackType="AnnotationTrack", start="chromStart",
                        end="chromEnd", id="name", shape="box",
                        fill="#006400", name="CpG",
refGenes <- UcscTrack(genome=genome, chromosome=chr,
                      track="refGene", from=start, to=end,
                      trackType="GeneRegionTrack", rstarts="exonStarts",
                      rends="exonEnds", gene="name2", symbol="name2",
                      transcript="name", strand="strand", fill="#8282d2",
                      name="refSeq Genes", collapseTranscripts=TRUE,
                      showId=TRUE, background.title="brown")
ensembl <- useMart(biomart="ENSEMBL_MART_ENSEMBL",
ensembl <-  useDataset("drerio_gene_ensembl",mart=ensembl)
biomTrack <- BiomartGeneRegionTrack(genome=genome, chromosome=chr, 
                                    start=start , end=end, name="Ensembl Genes")
plotTracks(list(axisTrack, cpgIslands, refGenes), 
           collapseTranscripts=TRUE, shape="arrow",

It is also possible to plot the annotation from an ordinary R object, such as data.frame, GRanges, IRanges or even from a local file. Usually the gff file containing the gene annotation can be processed by r Biocpkg("Gviz") directly. For more details, please look into the vignette of r Biocpkg("Gviz").

CNEs horizon plot

dbName <- file.path(system.file("extdata", package="CNEr"),
genome <- "danRer10"
windowSize <- 200L
minLength <- 50L
cneDanRer10Hg38_21_30 <- 
             whichAssembly="first", chr=chr, start=start,
             end=end, windowSize=windowSize, 
cneDanRer10Hg38_45_50 <-
             whichAssembly="first", chr=chr, start=start,
             end=end, windowSize=windowSize, 
cneDanRer10Hg38_49_50 <-
             whichAssembly="first", chr=chr, start=start,
             end=end, windowSize=windowSize, 
cneDanRer10AstMex102_48_50 <-
             whichAssembly="second", chr=chr, start=start,
             end=end, windowSize=windowSize, 
cneDanRer10CteIde1_75_75 <-
             whichAssembly="second", chr=chr, start=start,
             end=end, windowSize=windowSize, 
dTrack1 <- DataTrack(range=cneDanRer10Hg38_21_30,
                     genome=genome, type="horiz", 
                     fill.horizon=c("#B41414", "#E03231", "#F7A99C", 
                                    "yellow", "orange", "red"), 
                     name="human 21/30", background.title="brown")
dTrack2 <- DataTrack(range=cneDanRer10Hg38_45_50,
                     genome=genome, type="horiz", 
                     fill.horizon=c("#B41414", "#E03231", "#F7A99C", 
                                    "yellow", "orange", "red"), 
                     name="human 45/50", background.title="brown")
dTrack3 <- DataTrack(range=cneDanRer10Hg38_49_50,
                     genome=genome, type="horiz", 
                     fill.horizon=c("#B41414", "#E03231", "#F7A99C", 
                                    "yellow", "orange", "red"), 
                     name="human 49/50", background.title="brown")
dTrack4 <- DataTrack(range=cneDanRer10AstMex102_48_50,
                     genome=genome, type="horiz", 
                     fill.horizon=c("#B41414", "#E03231", "#F7A99C", 
                                    "yellow", "orange", "red"), 
                     name="blind cave fish 48/50", background.title="brown")
dTrack5 <- DataTrack(range=cneDanRer10CteIde1_75_75,
                     genome=genome, type="horiz", 
                     fill.horizon=c("#B41414", "#E03231", "#F7A99C", 
                                    "yellow", "orange", "red"), 
                     name="grass carp 75/75", background.title="brown")
ht <- HighlightTrack(trackList=list(refGenes, dTrack5, dTrack4, 
                                    dTrack1, dTrack2, dTrack3), 
                     start=c(24200000, 25200000, 26200000), 
                     end=c(25100000, 26150000, 27000000),
                     chromosome =chr)
plotTracks(list(axisTrack, cpgIslands, ht),
           collapseTranscripts=TRUE, shape="arrow",
           from=24000000, to=27000000)

From this horizon plot and when comparing Zebrafish with Human as a reference genome, we notice that the genes barhl2, lmo4b and sox14 are surrounded by the density peaks of CNEs.


r Biocpkg("CNEr") efficiently identifies CNEs and handles the corresponding objects conveniently in R.
Horizon plot shows a superior dynamic range to the standard density plots, simultaneously revealing CNE clusters characterized by vastly different levels of sequence conservation. Such CNE density plots generated using precise locations of CNEs can be used to identify genes involved in developmental regulation, even for novel genes that are not yet annotated.

The following is the session info that generated this vignette:



Try the CNEr package in your browser

Any scripts or data that you put into this service are public.

CNEr documentation built on Nov. 8, 2020, 5:36 p.m.