status.md
In RETURN-project/makeDataCube: Prepare raster datasets for upscaling recovery indicators
Status report
Current status
makeDataCube is now working properly on Spider.- GitHub actions manages to install a minimal version of
makeDataCube.
- What do I mean with minimal? A version with the minimal amount of external dependencies in order to be installable and testable.
- I've increased the test coverage a bit.
- I've rewritten the parameter generator function from
Python into R.
- And now we can use lists instead of long lines of parameters.
Things that can still be improved
Functionality
- At the moment, the output of
makeDataCube is not transferred to dCache. This should be relatively quick to implement:
- Minimum: copy contents without overwriting.
- Better: minimum + concatenate queue files.
- Optimum: better + summarize log files.
- Some processes can be further optimized.
- For instance, the metadata
csv files could be downloaded only once instead of once per process. Anyways, this download is really fast.
- Some functions still use several parameters instead of lists.
Usability
- The installation cannot be fully automated.
gsutil requires interactive installation. Or at least, we didn't find an automated / scripted alternative.- The
.netrc file, which contains credentials for accessing different services, has to be generated interactively. Luckily, this has to be only once.
- The same is true for the
.boto and .laads files, but I am not sure if they are really needed.
- The package is heavy. A minimal installation takes around 45 minutes.
- Most of the time is invested in installing
R dependencies. Probably we do not need all of them.
- Some of the functions can be further simplified.
- The workflow is intrinsically very complex. The main danger I see is that it depends on a lot of external services that cannot be frozen into a
Singularity image. I am afraid this package could be challenging to maintain in the long term.
RETURN-project/makeDataCube documentation built on Feb. 11, 2022, 3:04 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Status report
Current status
makeDataCubeis now working properly onSpider.- GitHub actions manages to install a minimal version of
makeDataCube. - What do I mean with minimal? A version with the minimal amount of external dependencies in order to be installable and testable.
- I've increased the test coverage a bit.
- I've rewritten the parameter generator function from
PythonintoR. - And now we can use lists instead of long lines of parameters.
Things that can still be improved
Functionality
- At the moment, the output of
makeDataCubeis not transferred todCache. This should be relatively quick to implement: - Minimum: copy contents without overwriting.
- Better: minimum + concatenate queue files.
- Optimum: better + summarize log files.
- Some processes can be further optimized.
- For instance, the metadata
csvfiles could be downloaded only once instead of once per process. Anyways, this download is really fast. - Some functions still use several parameters instead of lists.
Usability
- The installation cannot be fully automated.
gsutilrequires interactive installation. Or at least, we didn't find an automated / scripted alternative.- The
.netrcfile, which contains credentials for accessing different services, has to be generated interactively. Luckily, this has to be only once. - The same is true for the
.botoand.laadsfiles, but I am not sure if they are really needed. - The package is heavy. A minimal installation takes around 45 minutes.
- Most of the time is invested in installing
Rdependencies. Probably we do not need all of them. - Some of the functions can be further simplified.
- The workflow is intrinsically very complex. The main danger I see is that it depends on a lot of external services that cannot be frozen into a
Singularityimage. I am afraid this package could be challenging to maintain in the long term.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.