In USGS-R/scipiper: Support functions for ushering data through a scientific workflow

knitr::opts_chunk$set(echo = TRUE, cache=FALSE, collapse=TRUE)

The following bullets describe problems or frustrations that scipiper users might encounter, along with some ideas for how to interpret and fix those issues. Each "pitfall" is presented in this format: Problem or symptom. Explanation. Solution.

General use of scipiper/remake

• Unnecessary re-runs of functions. scipiper and remake don't update their databases when you run a function outside of scipiper, even if the function creates the desired outputs. This may be frustrating if you've just run a long-running function outside of scipiper and are now attempting to use scipiper to build targets that depend on the function's output, because scipiper will insist on running that function again. Anytime you might want the output of a function to be considered complete, build the scipiper target rather than calling the function directly.

• Common debugging methods don't work. If you build a scipiper target that runs function A() to create target a, setting RStudio checkpoints or debug(A) won't let you debug function A when you call scmake('a'). Temporarily insert a call to browser() within the function instead.

• The GitHub repo is too big. You might be committing data files or large task remake files to the repo. Data files should generally be excluded from the repo; instead of backing up data using the repo, these files should be backed up in the shared cache, should be quick to generate locally, or should (worst case) be backed up manually. You can .gitignore data files by specifying common data extensions in .gitignore and/or by storing data files in */tmp/* folders and .gitignoring those folders. You may also choose to make exceptions, committing some data or configuration files that are very small and/or text-based, especially when their git diffs would be informative (e.g., when a file describes user settings or summarizes the shared status of a project build). In general, you should .gitignore most data files, the .remake/ folder, and (optionally) any task remake files.

Use of shared cache

• Teammates aren't getting updates about available intermediate data files. When sharing intermediate data files with a team using a shared cache, the status information about those files is contained in (1) an .ind file promising the existence of the data file in the cache, and (2) a build/status/*.yml file telling remake how to decide whether the .ind file is up to date. Always git commit the .ind and build/status/*.yml files corresponding to all shared data files.

• Teammates can't see updated objects. R objects can't be shared. To share objects, save them as files (e.g., .rds, .feather, or .tsv) and use the standard shared cache syntax.

• Seemingly unnecessary builds of objects or files that a teammate already built. Those files may lack corresponding .ind files (and objects always lack .ind files), which are the key to avoiding rebuilds. With a shared cache approach, every data file or object not represented by an .ind file must be built locally by everybody who needs to build any target downstream of that file or object. For every file that takes a long time or special software to build: (1) use file rather than object targets, (2) define a corresponding .ind file and make downstream targets depend on the .ind file rather than the data file, (3) push the file to the shared cache, and (4) commit the .ind file to the repository.

• There are more .ind files than I want to manage. Not every data file (or object) needs to be in the shared cache, and anything not in the shared cache is a candidate for also not having an .ind file. If (a) you're comfortable requiring everybody to build a file or object locally, or (b) there are no declared downstream dependencies of the file/object, then you can skip creating an .ind file for it. For example, you might have a task remake file whose execution (e.g., with loop_tasks()) depends on X and Y and produces a single output file Z. As long as the main remake file says that loop_tasks() creates Z.ind (not Z), you can have just one person run the entire task remake file, and your team can share the output, without using any .ind files within those tasks. Don't create .ind files for targets that don't need them. Just remember that anything without an .ind file must be rebuilt by anyone else trying to access that target.

Shared cache during development

• I am working on code that changes data in the shared cache but I'm not ready to commit those changes, and teammates are being forced to rebuild. As you are developing code for targets that alter the shared cache, you will often want to test code for those targets without (yet) sharing the new changes with teammates. When you've changed the shared cache but haven't yet committed the .ind. or build/status/*.yml files, other teammates who are concurrently building targets that depend on your target will be forced to rebuild the target because their .ind and build/status/*.yml files show that they are outdated.While developing for a shared-cache target, use options(scipiper.dry_put = TRUE) to prevent files built from being pushed to the shared cache. Remember to revert to options(scipiper.dry_put = FALSE) and force a rebuild when you are ready to push your changes to the code, .ind, and build/status/*.yml files.

Team with multiple operating systems and shared caches

• Unnecessary rebuilds of text files with shared cache. Text data files (e.g., .csv or .txt) may contain different line endings (EOLs) depending on which package and OS were used to write them. This is a problem for text-based files that are scipiper targets, because what appears to be up to date on the originating OS will appear to be out of date on another OS, triggering unnecessary rebuilds. Avoid base file writing (e.g., base::writeLines, utils::write.csv). Instead use readr for all file writing.

• Unnecessary rebuilds of .rds files containing sf objects with shared cache. It appears that some sf objects are stored differently by each OS; again, because the hash in the build/status/*.yml file will then differ from the hash on the teammate's computer, this could trigger unnecessary rebuilds. Note that in some cases this might be the result of using two slightly different versions of sf rather than two different OSes. Either be wary and try sf objects for intermediate data storage anyway (it might work), or save these objects in another format.

• Unnecessary rebuilds of targets that depend on input text files. A hand-edited text file such as a YAML configuration file may be given different white space or line endings by different OSes (or even just different R projects), leading to unnecessary rebuilds if one user makes and commits a trivial change to that file. When starting a project, configure a shared *.Rproj file to use Insert spaces for tab: Tab width = 2; Strip trailing horizontal whitespace when saving = TRUE; and Line ending conversion: Posix (LF). If a shared *.Rproj file is not an option, create a .gitattributes file declaring * text eol=lf instead.

• Different results depending on the operating system. Sometimes only one or two operating systems produce the results we want - for example, Macs often make nicer .png files than Windows PCs, and some packages or software are only available on one platform. Use a shared cache and coordinate so that someone with the appropriate operating system runs these targets for the team, pushing the results (and .ind and build/status/*.yml files) to the shared cache.

• Note: Many OS-related challenges could also be solved by using Docker for development.

USGS-R/scipiper documentation built on May 25, 2023, 8:47 a.m.