In sapfluxnet/sapfluxnetQC1: Package with functions related to sapfluxnet project data

In order to identify the state of the data in the Data folder a status file is present in all sites folders. In this file is located all the information relative to the different steps of data processing for the folder datasets.

Format

Chosen format is yaml and the integration with R is done by the yaml package (https://cran.r-project.org/web/packages/yaml/index.html).

This format is chosen by several reasons:

1. Easy integration with R
2. Actively developed
3. Already in use in other packages/processes (RMarkdown).

Status file characteristics

• Location:
  Each status file resides in the root of the site data folder. For example, for the XXX_XXX_XXX site, the location is Data/XXX_XXX_XXX/status.file

• Name:
  Each file is named by the site code plus _status.yaml to indicate the nature of the file

• Content:
  Content of the file is in the yaml format. See content section below.

File content

QC : Field indicating if Quality Check is already done. Two sub-fields:

+ DONE: logical indicating if QC is done
+ DATE: character indicating the date in which QC was done

LVL1 : Field indicating if data is stored in the Level_1 data folder. Several sub-fields:

+ STORED: logical indicating if Lvl1 data was stored
+ DATE: character indicating the date in which storage was done
+ TO_LVL2: character indicating if the site is ready to go to LVL2. Valid
  values are:
    - "FREEZE": Don't move to LVL2, there is work to do yet in LVL1
    - "READY": Ready to move to LVL2, but not moved yet
    - "DONE": Already in LVL2

LVL2 : Field indicating if data is stored in the Level_2 data folder. Several sub-fields:

+ STORED: logical indicating if Lvl2 data was stored
+ DATE: character indicating the date in which storage was done
+ STEP: character indicating the Lvl2 step in which the site is at the moment.
  Posible values are:
    - "WARN": Data as in Lvl1, only with the outliers and ranges flags
    - "REM": Preprocessed data, with outliers and ranges values substituted
    - "UNITS": Processed data, with all the unit conversion steps performed
+ TO_REM: character indicating if the site is ready to go to REM. Valid
  values are:
    - "FREEZE": Don't move to REM, there is work to do yeT
    - "READY": Ready to move to REM, but not moved yet
    - "DONE": Already in REM
+ TO_UNITS: character indicating if the site is ready to go to UNITS. Valid
  values are:
    - "FREEZE": Don't move to UNITS, there is work to do yeT
    - "READY": Ready to move to UNITS, but not moved yet
    - "DONE": Already in UNITS
+ AVAIL: character vector with the available sapflow units stored

Example

XXX_XXX_XXX_status.yaml

# XXX_XXX_XXX_status.yaml

QC:
  DONE: yes
  DATE: '2016-05-29'
LVL1:
  STORED: yes
  DATE: '2016-05-30'
  TO_LVL2: "FREEZE"
LVL2:
  STORED: no
  DATE: ~
  STEP: ~
  TO_REM: "FREEZE"
  TO_UNITS: "FREEZE"
  AVAIL: ~

Getting and setting statuses

There is two functions to retrieve and set the status of a site:

1. df_get_status. In charge of load the actual state of a site. Executing this function in the RStudio Server instance prints a list with the elements we saw before. See function help (?df_get_status) for more info.

2. df_set_status. This function allows modification of the status file by providing a list for each element to modify. See function help (?df_set_status) for more info.

Resetting a site

Sometimes, after completing needed manual changes, status must be resetted to allow performing the QC process again. Status can be retrieved and changed by the functions explained above, but doing only this can lead to errors. When resetting a site, not only the status file must be changed, also old data present in received_data and Data/si_code/Lvl_1 folders must be renamed to avoid conflicts in the new QC. So, in the case of a site reset, we need to use the df_reset_data_status, which is in charge of setting the new status, but also of renaming old data.

sapfluxnet/sapfluxnetQC1 documentation built on May 29, 2019, 1:50 p.m.