In PredictiveEcology/SpaDES.experiment: Simulation Experiments Within The SpaDES Ecosystem

Running a groups of simulations: the experiment function

Once a simulation is properly initialized and runs correctly with the spades function, it is likely time to do things like run replicates. There are two functions, experiment and experiment2 to do this. These both give opportunities to create simulation experiments, such as through varying parameters and modules. experiment is for simpler "fully factorial" cases, and experiment2 can take any simList class objects for the simulation experiment. With experiment, for example, you can pass alternative values to parameters in modules and the experiment will build a fully factorial experiment, with replication and run them. This function (as with POM and splitRaster) is parallel-aware and can either proceed with a cluster object (cl argument) or a cluster running in the background using the raster::beginCluster.

library(SpaDES.experiment)
tmpdir <- file.path(tempdir(), "experiment")

# Copy Example 5 here:
mySim <- simInit(
  times = list(start = 0.0, end = 2.0, timeunit = "year"),
  params = list(.globals = list(stackName = "landscape", burnStats = "nPixelsBurned")),
  modules = list("randomLandscapes", "fireSpread", "caribouMovement"),
  paths = list(modulePath = system.file("sampleModules", package = "SpaDES.core"),
               outputPath = tmpdir),
  # Save final state of landscape and caribou
  outputs = data.frame(objectName = c("landscape", "caribou"), stringsAsFactors = FALSE)
)

sims <- experiment(mySim, replicates = 2, .plotInitialTime = NA) # no plotting
attr(sims, "experiment")$expDesign # shows 2 replicates of same experiment

Caching at the experiment level

As with spades, an experiment can be cached.

At the experiment level

This functionality can be achieved within an experiment call. This can be done 2 ways, either: "internally" through the cache argument, which will cache each spades call; or, "externally" which will cache the entire experiment. If there are lots of spades calls, then the former will be slow as the simList will be digested once per spades call.

Using cache argument

system.time({
  sims1 <- experiment(mySim, replicates = 2, cache = TRUE, .plotInitialTime = NA)
})

# internal -- second time faster
system.time({
  sims2 <- experiment(mySim, replicates = 2, cache = TRUE, .plotInitialTime = NA)
})
all.equal(sims1, sims2)

Wrapping experiment with Cache

Here, the simList (and other arguments to experiment) is hashed once, and if it is found to be the same as previous, then the returned list of simList objects is recovered. This means that even a very large experiment, with many replicates and combinations of parameters and modules can be recovered very quickly. Here we show that you can output objects to disk, so the list of simList objects doesn't get too big. Then, when we recover it in the Cached version, all the files are still there, the list of simList objects is small, so very fast to recover.

# External
outputs(mySim) <- data.frame(objectName = "landscape")
system.time({
  sims3 <- Cache(experiment, mySim, replicates = 3, .plotInitialTime = NA,
                 clearSimEnv = TRUE)
})

The second time is way faster. We see the output files in the same location.

system.time({
  sims4 <- Cache(experiment, mySim, replicates = 3, .plotInitialTime = NA,
                 clearSimEnv = TRUE)
})
# test they are all equal
lapply(1:2, function(x) all.equal(sims3[[x]], sims4[[x]])) 

dir(outputPath(mySim), recursive = TRUE)

Notice that speed up can be enormous; in this case ~100 times faster.

Nested Caching

This is a continuation of the Nested Caching section in the iv-caching vignetted in the SpaDES.core package. - Imagine we have large model, with many modules, with replication and alternative module collections (e.g., alternative fire models) - To run this would have a nested structure with the following functions:

simInit --> many .inputObjects calls

experiment --> many spades calls --> many module calls --> many event calls --> many function calls

Lets say we start to introduce caching to this structure. We start from the "inner" most functions that we could imaging Caching would be useful. Lets say there are some GIS operations, like raster::projectRaster, which operates on an input shapefile. We can Cache the projectRaster call to make this much faster, since it will always be the same result for a given input raster.

If we look back at our structure above, we see that we still have LOTS of places that are not Cached. That means that the experiment call will still spawn many spades calls, which will still spawn many module calls, and many event calls, just to get to the one Cache(projectRaster) call which is Cached. This function will likely be called hundreds of times (because experiment runs the spades call 100 times due to replication). This is good, but Cache does take some time. So, even if Cache(projectRaster) takes only 0.02 seconds, calling it hundreds of times means maybe 4 seconds. If we are doing this for many functions, then this will be too slow.

We can start putting Cache all up the sequence of calls. Unfortunately, the way we use Cache at each of these levels is a bit different, so we need a slightly different approach for each.

Cache the experiment call

Cache(experiment)

This will assess the simList (the objects, times, modules, etc.) and if they are all the same, it will return the final list of simLists that came from the first experiment call. NOTE: because this can be large, it is likely that you want clearSimEnv = TRUE, and have all objects that are needed after the experiment call saved to disk. Any stochasticity/randomness inside modules will be frozen. This is likely ok if the objective is to show results in a web app (via shiny or otherwise) or another visualization about the experiment outputs, e.g., comparing treatments, once sufficient stochasticity has been achieved.

mySimListOut <- Cache(experiment, mySim, clearSimEnv = TRUE)

Cache the spades calls inside experiment

experiment(cache = TRUE)

This will cache each of the spades calls inside the experiment call. That means that there are as many cache events as there are replicates and experimental treatments, which, again could be a lot. Like caching the experiment call, stochasticity/randomness will be frozen. Note, one good use of this is when you are making iterative, incremental replication, e.g.,

mySimOut <- experiment(mySim, replicates = 5, cache = TRUE)

You decide after waiting 10 minutes for it to finish, that you need more replication. Rather than start from zero replicates, you can just pick up where you left off:

mySimOut <- experiment(mySim, replicates = 10, cache = TRUE)

This will only add 5 more replicates.

PredictiveEcology/SpaDES.experiment documentation built on Dec. 3, 2019, 4:59 p.m.