PipelineTools: Class definition for pipeline tools

In raveio: File-System Toolbox for RAVE Project

Class definition for pipeline tools

Description

Class definition for pipeline tools

Class definition for pipeline tools

Value

The value of the inputs, or a list if key is missing

The values of the targets

A PipelineResult instance if as_promise or async is true; otherwise a list of values for input names

An environment of shared variables

See type

A table of the progress

Nothing

A new pipeline object based on the path given

the saved file path

the data if file is found or a default value

Active bindings

settings_path

absolute path to the settings file

extdata_path

absolute path to the user-defined pipeline data folder

target_table

table of target names and their descriptions

result_table

summary of the results, including signatures of data and commands

pipeline_path

the absolute path of the pipeline

pipeline_name

the code name of the pipeline

Methods

Public methods

• PipelineTools$new()

• PipelineTools$set_settings()

• PipelineTools$get_settings()

• PipelineTools$read()

• PipelineTools$run()

• PipelineTools$eval()

• PipelineTools$shared_env()

• PipelineTools$python_module()

• PipelineTools$progress()

• PipelineTools$attach()

• PipelineTools$visualize()

• PipelineTools$fork()

• PipelineTools$with_activated()

• PipelineTools$clean()

• PipelineTools$save_data()

• PipelineTools$load_data()

• PipelineTools$clone()

---

Method new()

construction function

Usage

PipelineTools$new(
  pipeline_name,
  settings_file = "settings.yaml",
  paths = pipeline_root(),
  temporary = FALSE
)

Arguments

pipeline_name

name of the pipeline, usually in the pipeline 'DESCRIPTION' file, or pipeline folder name

settings_file

the file name of the settings file, where the user inputs are stored

paths

the paths to find the pipeline, usually the parent folder of the pipeline; default is pipeline_root()

temporary

whether not to save paths to current pipeline root registry. Set this to TRUE when importing pipelines from subject pipeline folders

---

Method set_settings()

set inputs

Usage

PipelineTools$set_settings(..., .list = NULL)

Arguments

..., .list

named list of inputs; all inputs should be named, otherwise errors will be raised

---

Method get_settings()

get current inputs

Usage

PipelineTools$get_settings(key, default = NULL, constraint)

Arguments

key

the input name; default is missing, i.e., to get all the settings

default

default value if not found

constraint

the constraint of the results; if input value is not from constraint, then only the first element of constraint will be returned.

---

Method read()

read intermediate variables

Usage

PipelineTools$read(var_names, ifnotfound = NULL, ...)

Arguments

var_names

the target names, can be obtained via x$target_table member; default is missing, i.e., to read all the intermediate variables

ifnotfound

variable default value if not found

...

other parameters passing to pipeline_read

---

Method run()

run the pipeline

Usage

PipelineTools$run(
  names = NULL,
  async = FALSE,
  as_promise = async,
  scheduler = c("none", "future", "clustermq"),
  type = c("smart", "callr", "vanilla"),
  envir = new.env(parent = globalenv()),
  callr_function = NULL,
  return_values = TRUE,
  ...
)

Arguments

names

pipeline variable names to calculate; default is to calculate all the targets

async

whether to run asynchronous in another process

as_promise

whether to return a PipelineResult instance

scheduler, type, envir, callr_function, return_values, ...

passed to pipeline_run if as_promise is true, otherwise these arguments will be passed to pipeline_run_bare

---

Method eval()

run the pipeline in order; unlike $run(), this method does not use the targets infrastructure, hence the pipeline results will not be stored, and the order of names will be respected.

Usage

PipelineTools$eval(names, env = parent.frame(), clean = TRUE)

Arguments

names

pipeline variable names to calculate; must be specified

env

environment to evaluate and store the results

clean

whether to evaluate without polluting env

---

Method shared_env()

run the pipeline shared library in scripts starting with path R/shared

Usage

PipelineTools$shared_env()

---

Method python_module()

get 'Python' module embedded in the pipeline

Usage

PipelineTools$python_module(
  type = c("info", "module", "shared", "exist"),
  must_work = TRUE
)

Arguments

type

return type, choices are 'info' (get basic information such as module path, default), 'module' (load module and return it), 'shared' (load a shared sub-module from the module, which is shared also in report script), and 'exist' (returns true or false on whether the module exists or not)

must_work

whether the module needs to be existed or not. If TRUE, the raise errors when the module does not exist; default is TRUE, ignored when type is 'exist'.

---

Method progress()

get progress of the pipeline

Usage

PipelineTools$progress(method = c("summary", "details"))

Arguments

method

either 'summary' or 'details'

---

Method attach()

attach pipeline tool to environment (internally used)

Usage

PipelineTools$attach(env)

Arguments

env

an environment

---

Method visualize()

visualize pipeline target dependency graph

Usage

PipelineTools$visualize(
  glimpse = FALSE,
  aspect_ratio = 2,
  node_size = 30,
  label_size = 40,
  ...
)

Arguments

glimpse

whether to glimpse the graph network or render the state

aspect_ratio

controls node spacing

node_size, label_size

size of nodes and node labels

...

passed to pipeline_visualize

---

Method fork()

fork (copy) the current pipeline to a new directory

Usage

PipelineTools$fork(path, filter_pattern = PIPELINE_FORK_PATTERN)

Arguments

path

path to the new pipeline, a folder will be created there

filter_pattern

file pattern to copy

---

Method with_activated()

run code with pipeline activated, some environment variables and function behaviors might change under such condition (for example, targets package functions)

Usage

PipelineTools$with_activated(expr, quoted = FALSE, env = parent.frame())

Arguments

expr

expression to evaluate

quoted

whether expr is quoted; default is false

env

environment to run expr

---

Method clean()

clean all or part of the data store

Usage

PipelineTools$clean(
  destroy = c("all", "cloud", "local", "meta", "process", "progress", "objects",
    "scratch", "workspaces"),
  ask = FALSE
)

Arguments

destroy, ask

see tar_destroy

---

Method save_data()

save data to pipeline data folder

Usage

PipelineTools$save_data(
  data,
  name,
  format = c("json", "yaml", "csv", "fst", "rds"),
  overwrite = FALSE,
  ...
)

Arguments

data

R object

name

the name of the data to save, must start with letters

format

serialize format, choices are 'json', 'yaml', 'csv', 'fst', 'rds'; default is 'json'. To save arbitrary objects such as functions or environments, use 'rds'

overwrite

whether to overwrite existing files; default is no

...

passed to saver functions

---

Method load_data()

load data from pipeline data folder

Usage

PipelineTools$load_data(
  name,
  error_if_missing = TRUE,
  default_if_missing = NULL,
  format = c("auto", "json", "yaml", "csv", "fst", "rds"),
  ...
)

Arguments

name

the name of the data

error_if_missing

whether to raise errors if the name is missing

default_if_missing

default values to return if the name is missing

format

the format of the data, default is automatically obtained from the file extension

...

passed to loader functions

---

Method clone()

The objects of this class are cloneable with this method.

Usage

PipelineTools$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

pipeline

raveio documentation built on July 26, 2023, 5:29 p.m.