value_function: Value Function

In pomdp: Infrastructure for Partially Observable Markov Decision Processes (POMDP)

Value Function

Description

Extracts the value function from a solved model. Extracts the alpha vectors describing the value function. This is similar to policy() which in addition returns the action prescribed by the solution.

Usage

value_function(model)

plot_value_function(
  model,
  projection = NULL,
  epoch = 1,
  ylim = NULL,
  legend = TRUE,
  col = NULL,
  lwd = 1,
  lty = 1,
  ...
)

Arguments

model	a solved POMDP or MDP.
projection	Sample in a projected belief space. See projection() for details.
epoch	the value function of what epoch should be plotted? Use 1 for converged policies.
ylim	the y limits of the plot.
legend	logical; add a legend?
col	potting colors.
lwd	line width.
lty	line type.
...	additional arguments are passed on to stats::line()'.

Details

Plots the value function of a POMDP solution as a line plot. The solution is projected on two states (i.e., the belief for the other states is held constant at zero). The value function can also be visualized using plot_belief_space().

Value

the function as a matrix with alpha vectors as rows.

Author(s)

Michael Hahsler

See Also

Other policy: estimate_belief_for_nodes(), optimal_action(), plot_belief_space(), plot_policy_graph(), policy_graph(), policy(), projection(), reward(), solve_POMDP(), solve_SARSOP()

Other POMDP: POMDP_accessors, POMDP(), plot_belief_space(), projection(), regret(), sample_belief_space(), simulate_POMDP(), solve_POMDP(), solve_SARSOP(), transition_graph(), update_belief(), write_POMDP()

Examples

data("Tiger")
sol <- solve_POMDP(model = Tiger)
sol

# value function for the converged solution
value_function(sol)

plot_value_function(sol, ylim = c(0,20))

## finite-horizon problem
sol <- solve_POMDP(model = Tiger, horizon = 3, discount = 1,
  method = "enum")
sol

# inspect the value function for all epochs 
value_function(sol)

plot_value_function(sol, epoch = 1, ylim = c(-5, 25))
plot_value_function(sol, epoch = 2, ylim = c(-5, 25))
plot_value_function(sol, epoch = 3, ylim = c(-5, 25))

## Not run: 
# using ggplot2 to plot the value function for epoch 3
library(ggplot2)
pol <- policy(sol)[[3]]
ggplot(pol) + 
 geom_segment(aes(x = 0, y = `tiger-left`, xend = 1, yend = `tiger-right`, color = action)) + 
 coord_cartesian(ylim = c(-5, 15)) + ylab("Reward") + xlab("Belief")

## End(Not run)

pomdp documentation built on Sept. 9, 2023, 1:07 a.m.