In choisy/nu:

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%",
  eval = FALSE
)

nu

Installation

# install.packages("remotes")
remotes::install_github("choisy/nu")
remotes::install_github("RBigData/remoter")

Packages

Everything below relies on the use of the remoter package. Load it:

library(remoter)

Running commands interactively on a remote server

The way things are set up now is that each of the NUC* servers has an R session that is running in server mode and that is you can access seamlessly from your local R session thanks to this remoter package. For example, interactively accessing the R session that is running on NUC1:

client(nu::c1)

From here (as indicated by the prompt >remoter), whatever R command you type will be run on the distant R session on the NUC1 server. The result of these commands are sent to your local screen. To exit the remote R session and come back to your local R session, type:

exit()

Running an R script

Convert your Rmardown into a R script:

knitr::purl("my_analysis.Rmd")

It could be a good idea to edit this R script to (1) remove any data visualization that you're not interested here (your script will run faster) and (2) add commands to save calculation outputs to disk (just for safety, otherwise your calculation results will be in the remote working space anyway). Ex:

saveRDS(output1, "output1.rds")
saveRDS(output2, "output2.rds")

Or, alternatively, in one command:

save(output1, output2, file = "outputs.rda")

Or this if you want to save all the workspace:

save.image("workspace.rda")

Then to run the script on the NUC1 remote server:

batch(nu::c1, file = "my_analysis.R")

From here you can kill your local R session and your script keeps running on the remote R session. Once the calculations are done, you can fetch the results back to your local computer with the s2c() function. First you need to reconnect with the server in interactive mode:

client(nu::c1)

And then you can seamlessly bring R objects from the remote R session to your local one:

s2c(output1)
s2c(output2)

You can use the function c2s() if you want to do the opposite. Once you are interactively logged to the remote server you can use lsc() and rmc() to list and delete files from your local working space. As simple as that. And to access NUC1, NUC2, NUC3, NUC4 and NUC5, simply use the shortcuts nu::c1, nu::c2, nu::c3, nu::c4 and nu::c5 respectively.

Finally, remember that each of these 5 NUC computers has 8 cores. Use the mclapply() function (a multi-core version of the lapply() base function that basically replaces loops) from the parallel package to parallelize your calculation across these 8 cores. And if you want to spread your calculation to the 5 x 8 = 40 CPUs that are available in total, then you have to make 5 scripts that split the calculation in five and run these 5 batches:

batch(nu::c1, file = "my_analysis1.R")
batch(nu::c2, file = "my_analysis2.R")
batch(nu::c3, file = "my_analysis3.R")
batch(nu::c4, file = "my_analysis4.R")
batch(nu::c5, file = "my_analysis5.R")

You want to think of an R function that could seamlessly (1) generate the my_analysis1.R, my_analysis2.R, my_analysis3.R, my_analysis4.R and my_analysis5.R files from my_analysis.R and (2) call the 5 batch() commands.

choisy/nu documentation built on Feb. 22, 2021, 3:51 a.m.