README.md
In RETURN-project/makeDataCube: Prepare raster datasets for upscaling recovery indicators

makeDataCube

The makeDataCube R project generates a data cube from Landsat data using FORCE and allows to generate and add a data mask to the data cube.

You can install it via:

library(devtools)
install_github("RETURN-project/makeDataCube")

Install locally

This script requires certain programs to be installed in your computer in order to work. The guide below will help you through this process.

To download data from Google Cloud, gsutil should be installed.
Python 3 should be installed.
The pylandsat and shapely modules should be available to download data. Both can be installed installed via pip install pylandsat and pip install shapely.
In addition, FORCE (>= 3.6.3) should be installed. FORCE allows to generate a data cube of level-2 (or higher) Landsat and Sentinel-2 imagery from level-1 inputs. Please visit the project's website for more information and download instructions.
If the user wants to use parallelization of Level 1 download (by default this is deactivated), this (or a newer) version of FORCE is required. More information here.
Other system requirements are libpoppler-cpp-dev, libgdal-dev and libudunits2-dev. All of them can be installed via sudo apt-get install <library name>.

Install using a Docker image

We made available a Docker image with all the requirements pre-installed. Please run

docker pull returnproject/makedatacube:v01

to obtain it and add it to your docker images. After installing, you can use it, for example, like this:

docker run -t returnproject/makedatacube:v01 \
Rscript -e "rmarkdown::render('vignettes/make_Landsat_cube.Rmd', params = list(starttime = '2000-11-1', endtime = '2001-5-28’))”

The corresponding Dockerfile is also available in inst/docker/Dockerfile.

Keep in mind that these images cannot and should not include your credentials. Click here for a shortcut to include your credentials in your own Docker image.

Install using a Singularity image

If you are more comfortable with Singularity, we pushed an image to Singularity Cloud Services. You can use:

singularity pull --arch amd64 library://returnproject/default/makedatacube:v1

to obtain it. It will download a file with a name like makeDataCube_v01.sif. After installing, you can use it, for example, like this:

singularity exec makeDataCube_v01.sif \
Rscript -e "rmarkdown::render('vignettes/make_Landsat_cube.Rmd', params = list(starttime = '2000-11-1', endtime = '2001-5-28’))”

The corresponding .def file is available in inst/singularity/full.def.

Keep in mind that these images cannot and should not include your credentials. Click here for a shortcut to include your credentials in your own Singularity image.

This script connects to different data sources. Some of them require credentials, typically a username and a password. Please follow this guide to get yours:

gsutil should be authorized. This can be achieved by running gsutil config and following the instructions. The result of the activation is a .boto file.
The user should have a NASA Earthdata account to download DEM data. The Login, Username and Password are stored in a netrc file in the home directory. If no netrc file is found, you will be asked to provide your Username and Password and a netrc file will automatically be created (and stored for a next session). If you don't have an account yet, you can create one here. Afterwards, you can easily create the .netrc file by executing makeDataCube::EartDataLogin() in your R console and following the instructions.
Finally, you need authentication to download data from the LAADS DAAC (WVP data). To that end, you need an create a .laads file is in your home directory with a an App Key. The App Key can be requested from NASA Earthdata. This key should be stored in a file .laads in your home directory.

A presentation about this project was given at Wageningen University on Nov 8th 2021. The slides are temporarily available here.

If you experience any problem, please take a look at our troubleshooting section.

For launching the analysis in Spider follow the next steps:

Navigate to your personal space.
Clone this project git clone https://github.com/RETURN-project/makeDataCube.git.
Get inside the cloned directory: cd makeDataCube.
Edit inputs.csv with your desired inputs.
If desired, the processing parameter can be adjusted in the vignette make_Landsat_cube.Rmd. More information about these parameters can be found in the documentation and tutorials of FORCE.
Run sbatch --array=2-3 -N1 ./main.sh inputs.csv (note that --array=2-3 processes the lines 2 and 3 of inputs.csv).
The outputs will be transferred to OUTPUTD="/home/${USER}/outputs". Do you want to change this? Edit it in run.sh.

inputs.csv contains the data ranges to be processed. Each line would be send to a node.
run.sh takes care of the fine details.
main.sh calls run.sh in a loop, once per line in inputs.csv.

When running on Spider, the workflow is slightly modified: