This vignette will cover how to implement model pipelines using the rsyncrosim
package within the SyncroSim software framework. For an overview of SyncroSim and rsyncrosim
, as well as a basic usage tutorial for rsyncrosim
, see the Introduction to rsyncrosim
vignette. To learn how to use iterations in the rsyncrosim
interface, see the rsyncrosim
: introduction to uncertainty vignette.
helloworldPipeline
To demonstrate how to link models in a pipeline using the rsyncrosim
interface, we will need the helloworldPipeline SyncroSim package. helloworldPipeline
was designed to be a simple package to introduce pipelines to SyncroSim modeling workflows. Models (i.e. transformers) connected by pipelines allow the user to implement multiple transformers in a modeling workflow and access intermediate outputs of a transformer without having to create multiple scenarios.
The package takes from the user 3 inputs, mMean, mSD, and b. For each iteration, a value m, representing the slope, is sampled from a normal distribution with mean of mMean and standard deviation of mSD. The b value represents the intercept. In the first model in the pipeline, these input values are run through a linear model, y=mt+b, where t is time, and the y value is returned as output. The second model takes y as input and calculates the cumulative sum of y over time, returning a new variable yCum as output.
{width=600px}
For more details on the different features of the helloworldPipeline
SyncroSim package, consult the SyncroSim Enhancing a Package: Linking Models tutorial.
Before using rsyncrosim
you will first need to download and install the SyncroSim software. Versions of SyncroSim exist for both Windows and Linux.
Note: this tutorial was developed using rsyncrosim
version 2.0. To use rsyncrosim
version 2.0 or greater, SyncroSim version 3.0 or greater is required.
You will need to install the rsyncrosim
R package, either using CRAN or from the rsyncrosim
GitHub repository. Versions of rsyncrosim
are available for both Windows and Linux.
In a new R script, load the rsyncrosim
package.
# Load R package for working with SyncroSim library(rsyncrosim)
session()
Finish setting up the R environment for the rsyncrosim
workflow by creating a SyncroSim Session object. Use the session()
function to connect R to your installed copy of the SyncroSim software.
mySession <- session("path/to/install_folder") # Create a Session based SyncroSim install folder mySession <- session() # Using default install folder (Windows only) mySession # Displays the Session object
# Results of this code shown for above mySession <- session() # Using default install folder (Windows only) mySession # Displays the Session object
Use the version()
function to ensure you are using the latest version of SyncroSim.
version(mySession)
installPackage()
Install helloworldPipeline
using the rynscrosim
function installPackage()
. This function takes a package name as input and then queries the SyncroSim package server for the specified package.
installedPackages <- packages() if (is.element( "helloworldPipeline", installedPackages$name)) uninstallPackage( "helloworldPipeline")
# Install helloworldPipeline installPackage("helloworldPipeline")
helloworldPipeline
should now be included in the package list returned by the packages()
function in rsyncrosim
:
# Get list of installed packages packages()
installedPackages <- packages() pipeline_pkg <- installedPackages[installedPackages$name == "helloworldPipeline", ] row.names(pipeline_pkg) <- NULL pipeline_pkg
When creating a new modeling workflow from scratch, we need to create objects of the following scopes:
For more information on these scopes, see the Introduction to rsyncrosim
vignette.
if (file.exists("helloworldLibrary.ssim")){ deleteLibrary("helloworldLibrary.ssim", force = TRUE) }
# Create a new library myLibrary <- ssimLibrary(name = "helloworldLibrary.ssim", session = mySession, packages = "helloworldPipeline", overwrite = TRUE) # Open the default project myProject = project(ssimObject = myLibrary, project = "Definitions") # Create a new scenario (associated with the default project) myScenario = scenario(ssimObject = myProject, scenario = "My first scenario")
datasheet()
View the datasheets associated with your new scenario using the datasheet()
function from rsyncrosim
.
# View all datasheets associated with a library, project, or scenario datasheet(myScenario)
From the list of datasheets above, we can see that there are four datasheets specific to the helloworldPipeline
package, including an Inputs
datasheet, an Intermediate Outputs
datasheet, and an Outputs
datasheet. These three datasheets are connected by transformers. The values from the Inputs
datasheet are used as the input for the first transformer, which transforms the input data to output data through a series of model calculations. The output data from the first transformer is contained within the Intermediate Outputs
datasheet. The values from the Intermediate Outputs
datasheet are then used as input for the second transformer. The output from the second transformer is stored in the Outputs
datasheet.
datasheet()
and addRow()
Currently our input scenario datasheets are empty! We need to add some values to our Inputs
datasheet (InputDatasheet
) and Run Control datasheet (RunControl
) so we can run our model. We also need to add some information to the core Pipeline
datasheet to specify which transformers are run in which order.
Inputs Datasheet
First, assign the contents of the Inputs
datasheet to a new data frame variable using datasheet()
, then check the columns that need input values.
# Load Inputs datasheet to a new R data frame myInputDataframe <- datasheet(myScenario, name = "helloworldPipeline_InputDatasheet") # Check the columns of the input data frame str(myInputDataframe)
The Inputs
datasheet requires three values:
mMean
: the mean of the slope normal distribution.mSD
: the standard deviation of the slope normal distribution.b
: the intercept of the linear equation.Add these values to a new data frame, then use the addRow()
function from rsyncrosim
to update the input data frame
# Create input data and add it to the input data frame myInputRow <- data.frame(mMean = 2, mSD = 4, b = 3) myInputDataframe <- addRow(myInputDataframe, myInputRow) # Check values myInputDataframe
Finally, save the updated R data frame to a SyncroSim datasheet using saveDatasheet()
.
# Save input R data frame to a SyncroSim datasheet saveDatasheet(ssimObject = myScenario, data = myInputDataframe, name = "helloworldPipeline_InputDatasheet")
Run Control Datasheet
The Run Control
datasheet provides information about how many time steps and iterations to use in the model. Here, we set the number of iterations, as well as the minimum and maximum time steps for our model. Let's take a look at the columns that need input values.
# Load Run Control datasheet to a new R data frame runSettings <- datasheet(myScenario, name = "helloworldPipeline_RunControl") # Check the columns of the Run Control data frame str(runSettings)
The Run Control datasheet requires the following 3 columns:
MaximumIteration
: total number of iterations to run the model for.MinimumTimestep
: the starting time point of the simulation.MaximumTimestep
: the end time point of the simulation.Note: A fourth hidden column, MinimumIteration
, also exists in the Run Control datasheet (default=1).
We'll add this information to a new data frame and then add it to the Run Control data frame using addRow()
.
# Create Run Control data and add it to the Run Control data frame runSettingsRow <- data.frame(MaximumIteration = 5, MinimumTimestep = 1, MaximumTimestep = 10) runSettings <- addRow(runSettings, runSettingsRow) # Check values runSettings
Finally, save the R data frame to a SyncroSim datasheet using saveDatasheet()
.
# Save Run Control R data frame to a SyncroSim datasheet saveDatasheet(ssimObject = myScenario, data = runSettings, name = "helloworldPipeline_RunControl")
Pipeline Datasheet
To implement pipelines in our package, we need to specify the order in which to run the transformers in our pipeline by adding data to the Pipeline
datasheet. The Pipeline
datasheet is a built-in SyncroSim datasheet, meaning that it comes with every SyncroSim library regardless of which packages that library uses. We access it using the "core_" prefix with the datasheet()
function.
From viewing the structure of the Pipeline
datasheet we know that the StageNameId
is a factor with two levels:
We will set the data for this datasheet such that Hello World Pipeline 1 (R)
is run first, then Hello World Pipeline 2 (R)
. This way, the output from Hello World Pipeline 1 (R)
is used as the input for Hello World Pipeline 2 (R)
.
# Load Pipeline datasheet to a new R data frame myPipelineDataframe <- datasheet(myScenario, name = "core_Pipeline") # Check the columns of the Pipeline data frame str(myPipelineDataframe) # Create Pipeline data and add it to the Pipeline data frame myPipelineRow <- data.frame(StageNameId = c("Hello World Pipeline 1 (R)", "Hello World Pipeline 2 (R)"), RunOrder = c(1, 2)) myPipelineDataframe <- addRow(myPipelineDataframe, myPipelineRow) # Check values myPipelineDataframe # Save Pipeline R data frame to a SyncroSim datasheet saveDatasheet(ssimObject = myScenario, data = myPipelineDataframe, name = "core_Pipeline")
run()
We will now run our scenario using the run()
function in rsyncrosim
.
If we have a large model and we want to parallelize the run using multiprocessing, we can modify the library-scoped "core_Multiprocessing" datasheet. Since we are using five iterations in our model, we will set the number of jobs to five so each multiprocessing core will run a single iteration.
# Load list of available library-scoped datasheets datasheet(myLibrary) # Load the library-scoped multiprocessing datasheet multiprocess <- datasheet(myLibrary, name = "core_Multiprocessing") # Check required inputs str(multiprocess) # Enable multiprocessing multiprocess$EnableMultiprocessing <- TRUE # Set maximum number of jobs to 5 multiprocess$MaximumJobs <- 5 # Save multiprocessing configuration saveDatasheet(ssimObject = myLibrary, data = multiprocess, name = "core_Multiprocessing")
Now, when we run our scenario, it will use the desired multiprocessing configuration.
# Run the first scenario we created myResultScenario <- run(myScenario)
Once the run is complete, we can compare the original scenario to the result scenario to see which datasheets have been modified. Using the datasheet()
function with the optional
argument set to TRUE
, we see that data has been added to both the Intermediate Outputs
and Outputs
datasheets after running the scenario (see data
column below).
# Datasheets for original scenario datasheet(myScenario, optional = TRUE) # Datasheets for result scenario datasheet(myResultScenario, optional = TRUE)
The next step is to view the output datasheets added to the result scenario when it was run.
datasheet()
First, we will view the Intermediate Outputs
datasheet from the result scenario. We can load the result tables using the datasheet()
function. The Intermediate Outputs
datasheet corresponds to the results from the Hello World Pipeline 1
transformer stage.
# Results of first scenario resultsSummary <- datasheet(myResultScenario, name = "helloworldPipeline_IntermediateDatasheet") # View results table head(resultsSummary)
We can see that for every timestep in an iteration we have a new value of y corresponding to y=mt+b.
datasheet()
Now, we will view the final output datasheet from the result scenario. Again, we will use datasheet()
to load the result table. The Outputs
datasheet corresponds to the results from the Hello World Pipeline 2
transformer stage.
# Results of first scenario resultsSummary <- datasheet(myResultScenario, name = "helloworldPipeline_OutputDatasheet") # View results table head(resultsSummary)
We can see for each timestep in an iteration, we have a new value of yCum, representing the cumulative value of y over time.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.