README.md

In meterds/mdsrocker: Docker images for CI/CD and dockerized AWS Solutions at METER Data Science

mdsrocker

The goal of mdsrocker is to provide a framework for building a stack of version-stable docker images based on rocker/r-ver, an ubuntu-based docker image containing version-stable base R & a set of src build tools.

We follow the version tag approach of the rocker-project, using the image tag for specifying which version of R is desired (e.g. rocker/r-ver: 4.3.1). Omit the tag or specify :latest to always receive the latest versions.

Image stack

image description r-aws-minimal r-ver plus basic utilities stack including API tools r-aws-spatial r-aws-minimal plus a spatial libraries stack r-cicd-minimal r-aws-minimal plus a set of CI/CD tools r-cicd-spatial r-aws-spatial plus a set of CI/CD tools

Architecture

This building system is designed as an R package.

Data

All relevant information is stored in two data.frames (as package data):

• mdsrocker_installation: three different software bundles for the various scopes of the docker images, each containing predefined sets of software packages (system libraries, extra software packages, Python packages, R packages) to install.

type syslibs extra pypkgs rpkgs aws apt-utils , curl , git , jq , libbz2-dev , libpng-dev , libssl-dev , libsasl2-dev , libsodium-dev , python3-dev , python3-pip , python3-venv , software-properties-common AWS CLI version 2 pipreqs , poetry , requests arrow , checkmate , curl , dplyr , paws , plumber , renv , reticulate , targets , tarchetypes, units , xml2 , zoo spatial NULL WhiteboxTools NULL elevatr , gdalcubes , lwgeom , mapview , ncdf4 , openeo , raster , rstac , s2 , satellite , sf , sp , stars , starsExtra, terra , units , whitebox cicd NULL NULL NULL covr , DT , lintr , pkgdown , rcmdcheck, tinytest

• mdsrocker_dockerfiles: four dockerfile specifications, each containing the relevant information for generating the dockerfiles.

image parent script r-aws-minimal rocker/r-ver install_aws.sh r-aws-spatial meterds/r-aws-minimal install_spatial.sh r-cicd-minimal meterds/r-aws-minimal install_cicd.sh r-cicd-spatial meterds/r-aws-spatial install_cicd.sh

Scripts & Files

In this building system,

• standalone shell scripts containing all the install logic are located under the scripts/ directory,
• dockerfiles used in the Github Actions workflow to build the images are located under dockerfiles/, and
• two github actions workflows are stored under .github/workflows/:
• update-building-system.yml: re-builds all scripts if any changes were done in the repository (e.g. adding an R version to the DESCRIPTION file or adding some R packages to the software bundles),
• publish-docker-images.yml: actually builds and pushes the defined image stack.

Updating

All scripts are automatically updated in the Github Actions Workflow update-building-system.yml (by running execute.R) when commited and pushed.

For adding changes (e.g. additional R packages), do:

1. git pull the latest changes in the remote repository,
2. run:
   • mdseasy::easy_version(element = "dev", check_renv = FALSE, check_gitlab_ci = FALSE, push = FALSE)
3. update and run the code generating the package data in data-raw/datasets.R,
4. run:
   • devtools::document()
   • devtools::check()
5. update NEWS.md
6. run
   • gert::git_commit_all("YOUR COMMIT MESSAGE")
   • mdseasy::easy_version(check_renv = FALSE, check_gitlab_ci = FALSE)

Execution

The Github Actions workflow publish-docker-images.yml is:

• scheduled to run every Sunday,
• triggered by successful runs of the update-building-system.yml workflow on main branch,
• or can be started manually on Github.

meterds/mdsrocker documentation built on Dec. 2, 2023, 11:56 p.m.