Set up
library(learnr) library(PortfolioAnalytics) library(ROI) library(ROI.plugin.quadprog) library(tidyverse) library(tidyquant) library(ati) library(timetk) tutorial_options(exercise.eval = TRUE) knitr::opts_chunk$set(error = TRUE,warning=FALSE) library(foreach)
local set up
removals<-ls() rm(list=removals) # Always good practice to remove all object before beginning new work library(learnr) library(PortfolioAnalytics) library(ROI) library(ROI.plugin.quadprog) library(tidyverse) library(tidyquant) library(ati) library(timetk) tutorial_options(exercise.eval = TRUE) knitr::opts_chunk$set(error = TRUE, echo = TRUE,warning=FALSE, exercise=TRUE) library(foreach) registerDoSEQ()
Before you begin, run the following to update the packages. The install.packages command may not be necessary depending on your local
R libs.
remove.packages('ati')
remotes::install_github('barryquinn1/ati')
remotes::install_github('rstudio/learnr')
install.packages(c("foreach","iterators",'quadprog',"ROI","ROI.plugin.quadprog"))
.rs.restartR()
Welcome
In the previous exercises we considered a Marcenko-Pastur function which predefined the variance parameter. A more accuracy approach is to optimise the estimate of the Marcenko-Pastur distribution from the data underlying data then adjust the covariance matrix for the noise identified by this random matrix theory, the function estRMT() does exactly this.
estRMT algorithm
The main idea behind de-noising the covariance matrix is to eliminate the eigenvalues of the covariance matrix that are representing noise and not useful information.
Constant Residual Eigenvalue De-noising Method
This is done by determining the maximum theoretical value of the eigenvalue of such matrix as a threshold and then setting all the calculated eigenvalues above the threshold to the same value.
The function provided below for de-noising the covariance works as follows:
- The given covariance matrix is transformed to the correlation matrix.
- The eigenvalues and eigenvectors of the correlation matrix are calculated. *Using the Kernel Density Estimate algorithm a kernel of the eigenvalues is estimated.
- The Marcenko-Pastur pdf is fitted to the KDE estimate using the variance as the parameter for the optimization.
- From the obtained Marcenko-Pastur distribution, the maximum theoretical eigenvalue is calculated using the formula in Lopez de Prado (2016)
- The eigenvalues in the set that are above the theoretical value are all set to their average value.
For example, we have a set of 5 sorted eigenvalues $(\lambda_1, \lambda_2, ....., \lambda_5)$ two of which are above the maximum theoretical value, then we set $$\lambda_4^{NEW}=\lambda_5^{NEW}=\frac{\lambda_4^{OLD}+\lambda_5^{OLD}}{2}$$ Eigenvalues above the maximum theoretical value are left intact. $$\lambda_1^{NEW}=\lambda_1^{OLD}\\lambda_2^{NEW}=\lambda_2^{OLD}\\lambda_3^{NEW}=\lambda_3^{OLD}$$
- The new set of eigenvalues with the set of eigenvectors is used to obtain the new de-noised correlation matrix. $\tilde{C}$ is the de-noised correlation matrix, W is the eigenvectors matrix, and $\Lambda$ is the diagonal matrix with new eigenvalues.
- The new correlation matrix is then transformed back to the new de-noised covariance matrix.
$$\tilde{C}=W \Lambda W$$ * To rescale $\tilde{C}$ so that the main diagonal consists of 1s the following transformation is made. This is how the final $C_{denoised}$ is obtained.
$$C_{denoised}=\tilde{C}[(diag[\tilde{C}])^{1/2}(diag[\tilde{C}])^{1/2'}]^{-1}$$ * The new correlation matrix is then transformed back to the new de-noised covariance matrix.
Learning outcomes
In this tutorial, you will learn some portfolio analytics
- Preprocessing data for analysis
- Denoising real world data
- Detoning real world data
- Building and testing and investment strategy using the treated data
- Infering mean from the results
Topic 1 preprocessing the data
Pick out top 25 companies by market value from tsfe::FTS350data
A first step would be to rearrange the data to that price and market value are variables
# Data and covariance objects ftse350<-ati::ftse350 ftse350 %>% select(-Name) %>% spread(variable,value) %>% group_by(ticker) %>% summarise(mean_mv=mean(`Market Value`)) %>% mutate(rank = min_rank(desc(mean_mv))) %>% filter(rank<=25) %>% select(ticker) %>% unlist(use.names = F) -> tickers
ftse350 %>% select(-Name) %>% spread(variable,value) %>% group_by(ticker) %>% summarise(mean_mv=mean(`Market Value`)) %>% mutate(rank = min_rank(desc(mean_mv))) %>% filter(rank<=25) %>% select(ticker) %>% unlist(use.names = F) -> tickers
Create monthly log returns for each of the 25 stocks and reshape the data for random matrix theory analysis
ftse350 %>% select(-Name) %>% spread(variable,value) %>% filter(ticker %in% tickers) %>% group_by(ticker) %>% tq_transmute(select = Price, mutate_fun = monthlyReturn) %>% pivot_wider(names_from=ticker, values_from=monthly.returns)->ftse_r_m
Topic 2 Denoising
Estimating the theoretical Marcenko-Pastur distribution for real data
Hint: Look at the help file for the estRMT() function and estimate an denoised covariance matrix where the eigenvalues below the threshold are replaced with the average value.
ftse_r_m %>% tk_xts(silent = TRUE)->ftse_r_m_ts
model<-estRMT(ftse_r_m_ts)
Estimate the covariance of the ftse returns before the denoising and compare with the model results using heatmap()
ati::ftse25_rtns_mthly ->ftse_r_m
cov_denoise<-model$cov cov_raw<-cov(ftse_r_m[,-1]) heatmap(cov_raw); heatmap(cov_denoise)
As we can see, the main diagonal has not changed, but the other covariances are different. This means that the algorithm has affected those eigenvalues of the correlation matrix which have more noise associated with them.
Topic 3 Detoning
De-noised correlation matrix from the previous methods can also be de-toned by excluding a number of first eigenvectors representing the market component. According to Marcos Lopez de Prado (2020)
Financial correlation matrices usually incorporate a market component. The market component is characterized by the first eigenvector, with loadings $W_{n,1}\approx N^{-\frac{1}{2}}, n = 1, ..., N.$ Accordingly, a market component affects every item of the covariance matrix. In the context of clustering applications, it is useful to remove the market component, if it exists (a hypothesis that can be tested statistically). By removing the market component, we allow a greater portion of the correlation to be explained by components that affect specific subsets of the securities. It is similar to removing a loud tone that prevents us from hearing other sounds. The detoned correlation matrix is singular, as a result of eliminating (at least) one eigenvector. This is not a problem for clustering applications, as most approaches do not require the invertibility of the correlation matrix. Still, a detoned correlation matrix $C_{detoned}$ cannot be used directly for mean-variance portfolio optimization”*
The de-toning function works as follows:
- De-toning is applied on the de-noised correlation matrix.
- The correlation matrix representing the market component is calculated from market component eigenvectors and eigenvalues and then subtracted from the de-noised correlation matrix. This way the de-toned correlation matrix is obtained.
$$\hat{C}=C_{denoised} - W_m\Lambda_mW_m^{'}$$
- De-toned correlation matrix $\hat{C}$ is then rescaled so that the main diagonal consists of 1s
$$C_{detoned}=\hat{C}[(diag[\hat{C}])^{1/2}(diag[\hat{C}])^{1/2'}]^{-1}$$
One can apply de-toning to the covariance matrix by setting the detone parameter to True in estRMT function. Note that detoning will always be used with either of the denoising methods described before.
Detoning the previously denoised covariance and visualise a before and after comparison
ati::ftse25_rtns_mthly->ftse_r_m ftse_r_m %>% tk_xts(silent = TRUE)->ftse_r_m_ts model<-estRMT(ftse_r_m_ts) cov_denoise<-model$cov
model1<-estRMT(ftse_r_m_ts,detone = TRUE) model1$cov -> cov_detone heatmap(cov_denoise,main ="Simple Covariance");heatmap(cov_detone,main = "Detoned Covariance")
The results of de-toning are significantly different from the de-noising results. Notice how the axis labels have shifted. This indicates that the deleted market component had an effect on the covariance between elements.
Topic 4 backtest analysis of denoising
The major workhorse of this chapter is the portfolioAnalyticspackage developed by Peterson and Carl (2018).
Construct a custom moment function
Assume no third/fourth order effects.
custom.portfolio.moments <- function(R, portfolio) { momentargs<-list() momentargs$mu<-matrix(as.vector(apply(R,2, "mean")), ncol = 1) momentargs$sigma<-estRMT(R, parallel=FALSE)$cov momentargs$m3 <- matrix(0, nrow=ncol(R), ncol=ncol(R)^2) momentargs$m4 <- matrix(0, nrow=ncol(R), ncol=ncol(R)^3) return(momentargs) }
Portfolio set-up
Use the package PortfolioAnalytics we will construct a portfolio with the following specification.
1. No short sales are allowed.
2. All cash needs to be invested at all times.
3. Set the objective to maximize the quadratic utility which maximizes returns while controlling for risk.
pspec.lo <- portfolio.spec(assets = colnames(ftse_r_m_ts)) # Specification 1 and 2 pspec.lo <- add.constraint(pspec.lo, type="full_investment") pspec.lo <- add.constraint(pspec.lo, type="long_only") # Specification 3 pspec.lo <- add.objective(portfolio=pspec.lo, type="return", name="mean") pspec.lo <- add.objective(portfolio=pspec.lo, type="risk", name="var")
solver backend
library(ROI) library(ROI.plugin.quadprog) library(foreach) foreach::registerDoSEQ()
Backtest strategy analysis run
library(timetk) ati::ftse25_rtns_mthly->ftse_r_m ftse_r_m %>% tk_xts(silent = TRUE)->ftse_r_m_ts custom.portfolio.moments <- function(R, portfolio) { momentargs<-list() momentargs$mu<-matrix(as.vector(apply(R,2, "mean")), ncol = 1) momentargs$sigma<-estRMT(R, parallel=FALSE)$cov momentargs$m3 <- matrix(0, nrow=ncol(R), ncol=ncol(R)^2) momentargs$m4 <- matrix(0, nrow=ncol(R), ncol=ncol(R)^3) return(momentargs) } pspec.lo <- portfolio.spec(assets = colnames(ftse_r_m_ts)) # Specification 1 and 2 pspec.lo <- add.constraint(pspec.lo, type="full_investment") pspec.lo <- add.constraint(pspec.lo, type="long_only") # Specification 3 pspec.lo <- add.objective(portfolio=pspec.lo, type="return", name="mean") pspec.lo <- add.objective(portfolio=pspec.lo, type="risk", name="var")
registerDoSEQ() opt.ordinary <- optimize.portfolio.rebalancing( ftse_r_m_ts,pspec.lo, optimize_method="ROI", rebalance_on='months', training_period=30, trailing_periods=30) system.time( opt.rmt <-optimize.portfolio.rebalancing( ftse_r_m_ts, pspec.lo, optimize_method="ROI", momentFUN = "custom.portfolio.moments", rebalance_on="months", training_period=30, trailing_periods=30) )
Backtest results analysis
ati::ftse25_rtns_mthly->ftse_r_m ftse_r_m %>% tk_xts(silent = TRUE)->ftse_r_m_ts
ordinary.wts <- na.omit(extractWeights(opt.ordinary)) rmt.wts <- na.omit(extractWeights(opt.rmt)) ordinary <- Return.rebalancing(R=ftse_r_m_ts,weights=ordinary.wts) rmt <- Return.rebalancing(R=ftse_r_m_ts,weights=rmt.wts) rmt.strat.rets <- merge.xts(ordinary,rmt) colnames(rmt.strat.rets) <- c("ordinary", "rmt")
Chart the results
charts.PerformanceSummary(rmt.strat.rets, wealth.index = T, colorset = c("red","darkgrey"), main="Comparison of Portfolio Performance",cex.legend = 1.1,cex.axis = 1.1,legend.loc = "topleft")
Interpret your results
This is clear evidence that a denoised portfolio which is rebalanced monthly performs better than a portfolio which is optimised on the ordinary noisy returns. While both cumulative returns follow a similar time series path, the denoised portfolio returns experience much less drawdown than the ordinary returns portfolio.
References
Peterson, Brian G., and Peter Carl. 2018. PortfolioAnalytics: Portfolio Analysis, Including Numerical Methods for Optimization of Portfolios. https://github.com/braverock/PortfolioAnalytics.
