PolicyIterationRegular: Solve the optimal control problem using policy iteration,...

In Uffe-H-Thygesen/SDEtools: Utilities for Stochastic Differential Equations

Solve the optimal control problem using policy iteration, i.e. find the optimal strategy and value function for the case where the uncontrolled system is given by a subgenerator G0 (either due to discounting or due to absorbing boundaries)

Description

Solve the optimal control problem using policy iteration, i.e. find the optimal strategy and value function for the case where the uncontrolled system is given by a subgenerator G0 (either due to discounting or due to absorbing boundaries)

Usage

PolicyIterationRegular(
  G0,
  G1,
  k,
  uopt,
  iter.max = 1000,
  tol = 1e-12,
  do.minimize = TRUE,
  do.return.QSD = FALSE
)

Arguments

G0	The sub-generator of the uncontrolled system
G1	A list of (sub-)generators for each control
k	The running cost
uopt	A list of functions returning optional controls as function of
iter.max	= 1000 Maximum number of iterations
tol	= 1e-12 Tolerance for convergence
do.minimize=TRUE
do.return.QSD=FALSE	Compute and return the quasi-stationary distribution

Value

A list containing V: The value function, as a vector with an element for each state u: The optimal controls, as a matrix with a row for each state and a column for each control

and, if do.return.QSD==TRUE, qsd.value: The decay rate of the quasi-stationary distribution (decay rate) qsd.vector: The quasi-stationary distribution

Examples

## Controlling a system to the boundary with minimum effort
xi <- seq(-2,2,length=101)
xc <- as.numeric(cell.centers(xi,c(0,1))$x)
dx <- diff(xi)

G0 <- fvade(function(x)-x,function(x)1,xi,'a')
Gp <- fvade(function(x)1,function(x)0,xi,'a')
Gn <- fvade(function(x)-1,function(x)0,xi,'a')

uopt <- function(dV)pmax(0,-dV)
k <- function(u) 1 + 0.5*u[,1]^2 + 0.5*u[,2]^2
sol <- PolicyIterationRegular(G0,list(Gp,Gn),k,list(uopt,uopt),do.return.QSD=TRUE)

par(mfrow=c(1,3))
plot(xc,sol$V,xlab="x",ylab="Value function",type="l")
plot(xc,sol$u[,1]-sol$u[,2],type="l",xlab="x",ylab="Optimal control")
plot(xc,sol$qsd.vector/dx,type="l",xlab="x",ylab="QSD",main=sol$qsd.value)

Uffe-H-Thygesen/SDEtools documentation built on Jan. 15, 2025, 6:41 p.m.