PipelineTools: Class definition for pipeline tools
In beauchamplab/raveio: File-System Toolbox for RAVE Project
PipelineTools R Documentation
Class definition for pipeline tools
Description
Class definition for pipeline tools
Class definition for pipeline tools
Value
The value of the inputs, or a list if key is missing
The values of the targets
A PipelineResult instance if as_promise
or async is true; otherwise a list of values for input names
An environment of shared variables
See type
A table of the progress
Nothing
ancestor target names (including names)
A new pipeline object based on the path given
A new pipeline object based on the path given
the saved file path
the data if file is found or a default value
A list of key-value pairs
A list of the preferences. If simplify is true and length
if keys is 1, then returns the value of that preference
logical whether the keys exist
Active bindings
descriptionpipeline description
settings_pathabsolute path to the settings file
extdata_pathabsolute path to the user-defined pipeline data folder
preference_pathdirectory to the pipeline preference folder
target_tabletable of target names and their descriptions
result_tablesummary of the results, including
signatures of data and commands
pipeline_paththe absolute path of the pipeline
pipeline_namethe code name of the pipeline
Methods
Public methods
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Method new()
construction function
Usage
PipelineTools$new(
pipeline_name,
settings_file = "settings.yaml",
paths = pipeline_root(),
temporary = FALSE
)
Arguments
pipeline_namename of the pipeline, usually in the pipeline
'DESCRIPTION' file, or pipeline folder name
settings_filethe file name of the settings file, where the
user inputs are stored
pathsthe paths to find the pipeline, usually the parent folder
of the pipeline; default is pipeline_root()
temporarywhether not to save paths to current pipeline
root registry. Set this to TRUE when importing pipelines
from subject pipeline folders
Method set_settings()
set inputs
Usage
PipelineTools$set_settings(..., .list = NULL)
Arguments
..., .listnamed list of inputs; all inputs should be named,
otherwise errors will be raised
Method get_settings()
get current inputs
Usage
PipelineTools$get_settings(key, default = NULL, constraint)
Arguments
keythe input name; default is missing, i.e., to get all the
settings
defaultdefault value if not found
constraintthe constraint of the results; if input value is not
from constraint, then only the first element of constraint
will be returned.
Method read()
read intermediate variables
Usage
PipelineTools$read(var_names, ifnotfound = NULL, ...)
Arguments
var_namesthe target names, can be obtained via
x$target_table member; default is missing, i.e., to read
all the intermediate variables
ifnotfoundvariable default value if not found
...other parameters passing to pipeline_read
Method run()
run the pipeline
Usage
PipelineTools$run(
names = NULL,
async = FALSE,
as_promise = async,
scheduler = c("none", "future", "clustermq"),
type = c("smart", "callr", "vanilla"),
envir = new.env(parent = globalenv()),
callr_function = NULL,
return_values = TRUE,
...
)
Arguments
namespipeline variable names to calculate; default is to
calculate all the targets
asyncwhether to run asynchronous in another process
as_promisewhether to return a PipelineResult
instance
scheduler, type, envir, callr_function, return_values, ...passed to
pipeline_run if as_promise is true, otherwise
these arguments will be passed to pipeline_run_bare
Method eval()
run the pipeline in order; unlike $run(), this method
does not use the targets infrastructure, hence the pipeline
results will not be stored, and the order of names will be
respected.
Usage
PipelineTools$eval(
names,
env = parent.frame(),
shortcut = FALSE,
clean = TRUE,
...
)
Arguments
namespipeline variable names to calculate; must be specified
envenvironment to evaluate and store the results
shortcutlogical or characters; default is FALSE, meaning
names and all the dependencies (if missing from env)
will be evaluated; set to TRUE if only names are to be
evaluated. When shortcut is a character vector, it should be
a list of targets (including their ancestors) whose values can be assumed
to be up-to-date, and the evaluation of those targets can be skipped.
cleanwhether to evaluate without polluting env
...passed to pipeline_eval
Method shared_env()
run the pipeline shared library in scripts starting with
path R/shared
Usage
PipelineTools$shared_env(callr_function = callr::r)
Arguments
callr_functioneither callr::r or NULL; when
callr::r, the environment will be loaded in isolated R session
and serialized back to the main session to avoid contaminating the
main session environment; when NULL, the code will be sourced
directly in current environment.
Method python_module()
get 'Python' module embedded in the pipeline
Usage
PipelineTools$python_module(
type = c("info", "module", "shared", "exist"),
must_work = TRUE
)
Arguments
typereturn type, choices are 'info' (get basic information
such as module path, default), 'module' (load module and return
it), 'shared' (load a shared sub-module from the module, which
is shared also in report script), and 'exist' (returns true
or false on whether the module exists or not)
must_workwhether the module needs to be existed or not. If
TRUE, the raise errors when the module does not exist; default
is TRUE, ignored when type is 'exist'.
Method progress()
get progress of the pipeline
Usage
PipelineTools$progress(method = c("summary", "details"))
Arguments
methodeither 'summary' or 'details'
Method attach()
attach pipeline tool to environment (internally used)
Usage
PipelineTools$attach(env)
Arguments
envan environment
Method visualize()
visualize pipeline target dependency graph
Usage
PipelineTools$visualize(
glimpse = FALSE,
aspect_ratio = 2,
node_size = 30,
label_size = 40,
...
)
Arguments
glimpsewhether to glimpse the graph network or render the state
aspect_ratiocontrols node spacing
node_size, label_sizesize of nodes and node labels
...passed to pipeline_visualize
Method target_ancestors()
a helper function to get target ancestors
Usage
PipelineTools$target_ancestors(names, skip_names = NULL)
Arguments
namestargets whose ancestor targets need to be queried
skip_namestargets that are assumed to be up-to-date, hence
will be excluded, notice this exclusion is
recursive, that means not only skip_names are excluded,
but also their ancestors will be excluded from the result.
Method fork()
fork (copy) the current pipeline to a new directory
Usage
PipelineTools$fork(path, policy = "default")
Arguments
pathpath to the new pipeline, a folder will be created there
policyfork policy defined by module author, see text file
'fork-policy' under the pipeline directory; if missing, then default to
avoid copying main.html and shared folder
Method fork_to_subject()
fork (copy) the current pipeline to a 'RAVE' subject
Usage
PipelineTools$fork_to_subject(
subject,
label = "NA",
policy = "default",
delete_old = FALSE,
sanitize = TRUE
)
Arguments
subjectsubject ID or instance in which pipeline will be saved
labelpipeline label describing the pipeline
policyfork policy defined by module author, see text file
'fork-policy' under the pipeline directory; if missing, then default to
avoid copying main.html and shared folder
delete_oldwhether to delete old pipelines with the same label
default is false
sanitizewhether to sanitize the registry at save. This will
remove missing folders and import manually copied pipelines to the
registry (only for the pipelines with the same name)
Method with_activated()
run code with pipeline activated, some environment variables
and function behaviors might change under such condition (for example,
targets package functions)
Usage
PipelineTools$with_activated(expr, quoted = FALSE, env = parent.frame())
Arguments
exprexpression to evaluate
quotedwhether expr is quoted; default is false
envenvironment to run expr
Method clean()
clean all or part of the data store
Usage
PipelineTools$clean(
destroy = c("all", "cloud", "local", "meta", "process", "preferences", "progress",
"objects", "scratch", "workspaces"),
ask = FALSE
)
Arguments
destroy, asksee tar_destroy
Method save_data()
save data to pipeline data folder
Usage
PipelineTools$save_data(
data,
name,
format = c("json", "yaml", "csv", "fst", "rds"),
overwrite = FALSE,
...
)
Arguments
dataR object
namethe name of the data to save, must start with letters
formatserialize format, choices are 'json',
'yaml', 'csv', 'fst', 'rds'; default is
'json'. To save arbitrary objects such as functions or
environments, use 'rds'
overwritewhether to overwrite existing files; default is no
...passed to saver functions
Method load_data()
load data from pipeline data folder
Usage
PipelineTools$load_data(
name,
error_if_missing = TRUE,
default_if_missing = NULL,
format = c("auto", "json", "yaml", "csv", "fst", "rds"),
...
)
Arguments
namethe name of the data
error_if_missingwhether to raise errors if the name is missing
default_if_missingdefault values to return if the name is missing
formatthe format of the data, default is automatically obtained
from the file extension
...passed to loader functions
Method set_preferences()
set persistent preferences from the pipeline.
The preferences should not affect how pipeline is working, hence usually
stores minor variables such as graphic options. Changing preferences
will not invalidate pipeline cache.
Usage
PipelineTools$set_preferences(..., .list = NULL)
Arguments
..., .listkey-value pairs of initial preference values. The keys
must start with 'global' or the module ID, followed by dot and preference
type and names. For example 'global.graphics.continuous_palette'
for setting palette colors for continuous heat-map; "global" means the
settings should be applied to all 'RAVE' modules. The module-level
preference, 'power_explorer.export.default_format' sets the
default format for power-explorer export dialogue.
namepreference name, must contain only letters, digits,
underscore, and hyphen, will be coerced to lower case (case-insensitive)
Method get_preferences()
get persistent preferences from the pipeline.
Usage
PipelineTools$get_preferences(
keys,
simplify = TRUE,
ifnotfound = NULL,
validator = NULL,
...
)
Arguments
keyscharacters to get the preferences
simplifywhether to simplify the results when length of key is 1;
default is true; set to false to always return a list of preferences
ifnotfounddefault value when the key is missing
validatorNULL or function to validate the values; see
'Examples'
...passed to validator if validator is a function
Examples
library(raveio)
if(interactive() && length(pipeline_list()) > 0) {
pipeline <- pipeline("power_explorer")
# set dummy preference
pipeline$set_preferences("global.example.dummy_preference" = 1:3)
# get preference
pipeline$get_preferences("global.example.dummy_preference")
# get preference with validator to ensure the value length to be 1
pipeline$get_preferences(
"global.example.dummy_preference",
validator = function(value) {
stopifnot(length(value) == 1)
},
ifnotfound = 100
)
pipeline$has_preferences("global.example.dummy_preference")
}
Method has_preferences()
whether pipeline has preference keys
Usage
PipelineTools$has_preferences(keys, ...)
Arguments
keyscharacters name of the preferences
...passed to internal methods
Method clone()
The objects of this class are cloneable with this method.
Usage
PipelineTools$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
See Also
pipeline
Examples
## ------------------------------------------------
## Method `PipelineTools$get_preferences`
## ------------------------------------------------
library(raveio)
if(interactive() && length(pipeline_list()) > 0) {
pipeline <- pipeline("power_explorer")
# set dummy preference
pipeline$set_preferences("global.example.dummy_preference" = 1:3)
# get preference
pipeline$get_preferences("global.example.dummy_preference")
# get preference with validator to ensure the value length to be 1
pipeline$get_preferences(
"global.example.dummy_preference",
validator = function(value) {
stopifnot(length(value) == 1)
},
ifnotfound = 100
)
pipeline$has_preferences("global.example.dummy_preference")
}
beauchamplab/raveio documentation built on March 5, 2025, 11:06 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| PipelineTools | R Documentation |
Class definition for pipeline tools
Description
Class definition for pipeline tools
Class definition for pipeline tools
Value
The value of the inputs, or a list if key is missing
The values of the targets
A PipelineResult instance if as_promise
or async is true; otherwise a list of values for input names
An environment of shared variables
See type
A table of the progress
Nothing
ancestor target names (including names)
A new pipeline object based on the path given
A new pipeline object based on the path given
the saved file path
the data if file is found or a default value
A list of key-value pairs
A list of the preferences. If simplify is true and length
if keys is 1, then returns the value of that preference
logical whether the keys exist
Active bindings
descriptionpipeline description
settings_pathabsolute path to the settings file
extdata_pathabsolute path to the user-defined pipeline data folder
preference_pathdirectory to the pipeline preference folder
target_tabletable of target names and their descriptions
result_tablesummary of the results, including signatures of data and commands
pipeline_paththe absolute path of the pipeline
pipeline_namethe code name of the pipeline
Methods
Public methods
Method new()
construction function
Usage
PipelineTools$new( pipeline_name, settings_file = "settings.yaml", paths = pipeline_root(), temporary = FALSE )
Arguments
pipeline_namename of the pipeline, usually in the pipeline
'DESCRIPTION'file, or pipeline folder namesettings_filethe file name of the settings file, where the user inputs are stored
pathsthe paths to find the pipeline, usually the parent folder of the pipeline; default is
pipeline_root()temporarywhether not to save
pathsto current pipeline root registry. Set this toTRUEwhen importing pipelines from subject pipeline folders
Method set_settings()
set inputs
Usage
PipelineTools$set_settings(..., .list = NULL)
Arguments
..., .listnamed list of inputs; all inputs should be named, otherwise errors will be raised
Method get_settings()
get current inputs
Usage
PipelineTools$get_settings(key, default = NULL, constraint)
Arguments
keythe input name; default is missing, i.e., to get all the settings
defaultdefault value if not found
constraintthe constraint of the results; if input value is not from
constraint, then only the first element ofconstraintwill be returned.
Method read()
read intermediate variables
Usage
PipelineTools$read(var_names, ifnotfound = NULL, ...)
Arguments
var_namesthe target names, can be obtained via
x$target_tablemember; default is missing, i.e., to read all the intermediate variablesifnotfoundvariable default value if not found
...other parameters passing to
pipeline_read
Method run()
run the pipeline
Usage
PipelineTools$run(
names = NULL,
async = FALSE,
as_promise = async,
scheduler = c("none", "future", "clustermq"),
type = c("smart", "callr", "vanilla"),
envir = new.env(parent = globalenv()),
callr_function = NULL,
return_values = TRUE,
...
)Arguments
namespipeline variable names to calculate; default is to calculate all the targets
asyncwhether to run asynchronous in another process
as_promisewhether to return a
PipelineResultinstancescheduler, type, envir, callr_function, return_values, ...passed to
pipeline_runifas_promiseis true, otherwise these arguments will be passed topipeline_run_bare
Method eval()
run the pipeline in order; unlike $run(), this method
does not use the targets infrastructure, hence the pipeline
results will not be stored, and the order of names will be
respected.
Usage
PipelineTools$eval( names, env = parent.frame(), shortcut = FALSE, clean = TRUE, ... )
Arguments
namespipeline variable names to calculate; must be specified
envenvironment to evaluate and store the results
shortcutlogical or characters; default is
FALSE, meaningnamesand all the dependencies (if missing fromenv) will be evaluated; set toTRUEif onlynamesare to be evaluated. Whenshortcutis a character vector, it should be a list of targets (including their ancestors) whose values can be assumed to be up-to-date, and the evaluation of those targets can be skipped.cleanwhether to evaluate without polluting
env...passed to
pipeline_eval
Method shared_env()
run the pipeline shared library in scripts starting with
path R/shared
Usage
PipelineTools$shared_env(callr_function = callr::r)
Arguments
callr_functioneither
callr::rorNULL; whencallr::r, the environment will be loaded in isolated R session and serialized back to the main session to avoid contaminating the main session environment; whenNULL, the code will be sourced directly in current environment.
Method python_module()
get 'Python' module embedded in the pipeline
Usage
PipelineTools$python_module(
type = c("info", "module", "shared", "exist"),
must_work = TRUE
)Arguments
typereturn type, choices are
'info'(get basic information such as module path, default),'module'(load module and return it),'shared'(load a shared sub-module from the module, which is shared also in report script), and'exist'(returns true or false on whether the module exists or not)must_workwhether the module needs to be existed or not. If
TRUE, the raise errors when the module does not exist; default isTRUE, ignored whentypeis'exist'.
Method progress()
get progress of the pipeline
Usage
PipelineTools$progress(method = c("summary", "details"))Arguments
methodeither
'summary'or'details'
Method attach()
attach pipeline tool to environment (internally used)
Usage
PipelineTools$attach(env)
Arguments
envan environment
Method visualize()
visualize pipeline target dependency graph
Usage
PipelineTools$visualize( glimpse = FALSE, aspect_ratio = 2, node_size = 30, label_size = 40, ... )
Arguments
glimpsewhether to glimpse the graph network or render the state
aspect_ratiocontrols node spacing
node_size, label_sizesize of nodes and node labels
...passed to
pipeline_visualize
Method target_ancestors()
a helper function to get target ancestors
Usage
PipelineTools$target_ancestors(names, skip_names = NULL)
Arguments
namestargets whose ancestor targets need to be queried
skip_namestargets that are assumed to be up-to-date, hence will be excluded, notice this exclusion is recursive, that means not only
skip_namesare excluded, but also their ancestors will be excluded from the result.
Method fork()
fork (copy) the current pipeline to a new directory
Usage
PipelineTools$fork(path, policy = "default")
Arguments
pathpath to the new pipeline, a folder will be created there
policyfork policy defined by module author, see text file 'fork-policy' under the pipeline directory; if missing, then default to avoid copying
main.htmlandsharedfolder
Method fork_to_subject()
fork (copy) the current pipeline to a 'RAVE' subject
Usage
PipelineTools$fork_to_subject( subject, label = "NA", policy = "default", delete_old = FALSE, sanitize = TRUE )
Arguments
subjectsubject ID or instance in which pipeline will be saved
labelpipeline label describing the pipeline
policyfork policy defined by module author, see text file 'fork-policy' under the pipeline directory; if missing, then default to avoid copying
main.htmlandsharedfolderdelete_oldwhether to delete old pipelines with the same label default is false
sanitizewhether to sanitize the registry at save. This will remove missing folders and import manually copied pipelines to the registry (only for the pipelines with the same name)
Method with_activated()
run code with pipeline activated, some environment variables
and function behaviors might change under such condition (for example,
targets package functions)
Usage
PipelineTools$with_activated(expr, quoted = FALSE, env = parent.frame())
Arguments
exprexpression to evaluate
quotedwhether
expris quoted; default is falseenvenvironment to run
expr
Method clean()
clean all or part of the data store
Usage
PipelineTools$clean(
destroy = c("all", "cloud", "local", "meta", "process", "preferences", "progress",
"objects", "scratch", "workspaces"),
ask = FALSE
)Arguments
destroy, asksee
tar_destroy
Method save_data()
save data to pipeline data folder
Usage
PipelineTools$save_data(
data,
name,
format = c("json", "yaml", "csv", "fst", "rds"),
overwrite = FALSE,
...
)Arguments
dataR object
namethe name of the data to save, must start with letters
formatserialize format, choices are
'json','yaml','csv','fst','rds'; default is'json'. To save arbitrary objects such as functions or environments, use'rds'overwritewhether to overwrite existing files; default is no
...passed to saver functions
Method load_data()
load data from pipeline data folder
Usage
PipelineTools$load_data(
name,
error_if_missing = TRUE,
default_if_missing = NULL,
format = c("auto", "json", "yaml", "csv", "fst", "rds"),
...
)Arguments
namethe name of the data
error_if_missingwhether to raise errors if the name is missing
default_if_missingdefault values to return if the name is missing
formatthe format of the data, default is automatically obtained from the file extension
...passed to loader functions
Method set_preferences()
set persistent preferences from the pipeline. The preferences should not affect how pipeline is working, hence usually stores minor variables such as graphic options. Changing preferences will not invalidate pipeline cache.
Usage
PipelineTools$set_preferences(..., .list = NULL)
Arguments
..., .listkey-value pairs of initial preference values. The keys must start with 'global' or the module ID, followed by dot and preference type and names. For example
'global.graphics.continuous_palette'for setting palette colors for continuous heat-map; "global" means the settings should be applied to all 'RAVE' modules. The module-level preference,'power_explorer.export.default_format'sets the default format for power-explorer export dialogue.namepreference name, must contain only letters, digits, underscore, and hyphen, will be coerced to lower case (case-insensitive)
Method get_preferences()
get persistent preferences from the pipeline.
Usage
PipelineTools$get_preferences( keys, simplify = TRUE, ifnotfound = NULL, validator = NULL, ... )
Arguments
keyscharacters to get the preferences
simplifywhether to simplify the results when length of key is 1; default is true; set to false to always return a list of preferences
ifnotfounddefault value when the key is missing
validatorNULLor function to validate the values; see 'Examples'...passed to
validatorifvalidatoris a function
Examples
library(raveio)
if(interactive() && length(pipeline_list()) > 0) {
pipeline <- pipeline("power_explorer")
# set dummy preference
pipeline$set_preferences("global.example.dummy_preference" = 1:3)
# get preference
pipeline$get_preferences("global.example.dummy_preference")
# get preference with validator to ensure the value length to be 1
pipeline$get_preferences(
"global.example.dummy_preference",
validator = function(value) {
stopifnot(length(value) == 1)
},
ifnotfound = 100
)
pipeline$has_preferences("global.example.dummy_preference")
}
Method has_preferences()
whether pipeline has preference keys
Usage
PipelineTools$has_preferences(keys, ...)
Arguments
keyscharacters name of the preferences
...passed to internal methods
Method clone()
The objects of this class are cloneable with this method.
Usage
PipelineTools$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
See Also
pipeline
Examples
## ------------------------------------------------
## Method `PipelineTools$get_preferences`
## ------------------------------------------------
library(raveio)
if(interactive() && length(pipeline_list()) > 0) {
pipeline <- pipeline("power_explorer")
# set dummy preference
pipeline$set_preferences("global.example.dummy_preference" = 1:3)
# get preference
pipeline$get_preferences("global.example.dummy_preference")
# get preference with validator to ensure the value length to be 1
pipeline$get_preferences(
"global.example.dummy_preference",
validator = function(value) {
stopifnot(length(value) == 1)
},
ifnotfound = 100
)
pipeline$has_preferences("global.example.dummy_preference")
}
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.