In beauchamplab/raveio: File-System Toolbox for RAVE Project

# This code block sets up the engine environment
# Please do not remove me
raveio::pipeline_setup_rmd("TEMPLATE")

Introduction

A RAVE pipeline markdown is an interactive notebook that can keep your notes, code blocks, and corresponding results together, generating reports in various formats such as PDF, html, Word, PowerPoint.

The note parts are simply markdowns - the same as jupyter notebook, or github documentations. The code blocks support R, python, c++. When you hit the Knit button in this code editor panel, the r-markdown file will be compiled, generating reproducible documentation.

With carefully designed structures, this r-markdown file will automatically generate RAVE pipeline scripts during the compilation. The pipeline script can be used by RAVE to convert your pipeline into interactive dashboard application. (This feature is currently under development)

"RAVE" Pipeline Code Block

A RAVE pipeline markdown code block starts with ```{rave .... The block label following rave informative description of the target. After the target, the following RAVE-specific parameters configures how the block should be treated:

language: specifies the programming language used; choices are: R, python
export: variable name to be exported that will be available to the rest chunks
depends: indicates current block depends on variables generated from other blocks; this helps RAVE to build non-interactive pipeline scripts internally. For blocks written in R, the dependence can be automatically determined.

Other parameters are available at this rmarkdown book

An Example

In the rest of the documentation, let's import the subject power data, baseline, and plot the collapsed mean as image.

Step 1: Create `RAVE` subject's instances

Noting that all the items in the settings.yaml are available as variables.

library(raveio)
subject <- RAVESubject$new(
  project_name = project_name, 
  subject_code = subject_code
)
print(subject)

# List subject details
subject$epoch_names
subject$reference_names
subject$blocks
subject$electrodes

# Check if notch & wavelet has been applied on all electrodes
all(subject$notch_filtered)
all(subject$has_wavelet)

With export="subject", the subject variable will be registered for the following chunks to use. Be aware that all other variables created in this block will not be exposed.

Step 2: Load epoch and reference information

epoch_instance <- raveio::RAVEEpoch$new(subject, epoch)
head(epoch_instance$update_table())

reference_table <- subject$meta_data(
  meta_type = "reference", reference)
head(reference_table)

Step 3: Initialize and load power data

Initialize the electrode instances and register the epoch, reference information

requested_electrodes <- dipsaus::parse_svec(electrodes)
requested_electrodes <- requested_electrodes[
  requested_electrodes %in% subject$electrodes]
electrode_instances <- lapply(requested_electrodes, function(ei){
  e <- raveio::new_electrode(subject, number = ei, signal_type = "LFP")

  # set epoch
  e$set_epoch(epoch_instance)

  # set epoch range (-1 to 2 seconds relative to onset)
  e$trial_intervals <- intervals

  # set reference
  ref_name <- subset(reference_table, Electrode == ei)[["Reference"]]
  e$set_reference(ref_name)
  e
})
names(electrode_instances) <- requested_electrodes

Start to load power. Here also create cache to the RAVE cache directory.

power_list <- lapply(electrode_instances, function(e){
  e$load_data("power")
})
target <- normalizePath(file.path("shared", "cache", "power"),
                        mustWork = FALSE)

if(dir.exists(target)){
  unlink(target, recursive = TRUE, force = TRUE)
} else {
  raveio::dir_create2(dirname(target))
}
power <- filearray::filearray_bind(
  .list = power_list, symlink = TRUE, 
  filebase = target)

if(length(power_list)){
  dnames <- dimnames(power_list[[1]])
  dnames[[4]] <- as.integer(names(electrode_instances))
  dimnames(power) <- dnames
} else {
  stop("No power data not loaded")
}

Step 4: Baseline correction

dim <- dim(power)
dim[length(dim)] <- 1
plen <- prod(dim)
target <- file.path(electrode_instances[[1]]$cache_root, "..")
raveio::dir_create2(target)
target <- normalizePath(target, mustWork = TRUE)
target <- file.path(target, "_baseline")
if(dir.exists(target)){
  output <- filearray::filearray_load(target, "readwrite")
  if(!all(dim(output) == dim(power))){
    output$delete()
  }
  if(!identical(dimnames(output), dimnames(power))){
    dimnames(output) <- dimnames(power)
  }
}
if(!dir.exists(target)){
  output <- filearray::filearray_create(
    filebase = target,
    dimension = dim(power),
    type = "float",
    partition_size = 1L
  )
}

bl_range <- unlist(baseline)
dnames <- dimnames(power)
baseline_indexpoints <- which(
  dnames$Time >= bl_range[[1]] & dnames$Time <= bl_range[[2]])

filearray::fmap(power, function(input){
  x <- input[[1]]
  dim(x) <- dim
  bl <- dipsaus::baseline_array(
    x, along_dim = 2L,
    baseline_indexpoints = baseline_indexpoints,
    method = baseline_method
  )
  bl
}, .y = output, .input_size = plen)

baseline_data <- output

Step 5: Collapse baseline data

dim <- dim(baseline_data)
dim[length(dim)] <- 1
plen <- prod(dim)
collase_by_trial <- filearray::fmap2(baseline_data, function(input){
  x <- input[[1]]
  dim(x) <- dim
  dipsaus::collapse(x, keep = c(2,1), average = TRUE)
}, .input_size = plen, .simplify = TRUE)

collapsed <- list(
  collase_by_trial = collase_by_trial,
  collase_by_trial_electrodes = dipsaus::collapse(
    collase_by_trial, keep = c(1, 2), average = TRUE)
)

brain <- threeBrain::freesurfer_brain2(
  fs_subject_folder = subject$freesurfer_path, subject_name = subject$subject_code
)
brain$plot()

Step 6: Visualize

# Timestamp of when the plot is created
collase_by_trial_timestamp <- Sys.time()

# visualize
dnames <- power$dimnames()

image(
  collapsed$collase_by_trial_electrodes,
  y = dnames$Frequency,
  ylab = "Frequency (Hz)",
  x = dnames$Time, 
  xlab = "Time (s)",
  main = "Mean power across trial and electrodes"
)

Build, Visualize, & Run

Please make sure the following code block is at the end of your pipeline file. This block will build the pipeline and generate a make-TEMPLATE.R script with your pipeline markdown file. RAVE will use the generated pipeline script to execute the pipeline in the dashboard application, or in massive production mode.

build_pipeline(make_file = "make-TEMPLATE.R")

Once the pipeline script make-TEMPLATE.R is built, you can visualize and execute the pipeline without the need of re-knit this document. Notice we use r block instead of rave. (This is because the code blocks are not part of pipeline targets.)

# Fixed usage, show pipeline graph
try({
  asNamespace("raveio")$pipeline_dependency_graph(
    pipeline_path = ".", glimpse = TRUE)
}, silent = TRUE)