process_cdom_data: Process CDOM data from Excel sheets

In MikkoVihtakari/MarineDatabase: Export data to Norwegian Polar Institute's marine database

Description

Reads CDOM data from Excel sheets and calculates a range of parameters that can be exported to the database.

Usage

process_cdom_data(data_file, sheet = "all", blank_correction = "One milliQ")

Arguments

data_file	Name of the excel file containing the dataset to be exported.
sheet	sheet number or name where data are located. If "all" (default), all sheets in the Excel file are opened and processed. See Details and read.xlsx for further information.
blank_correction	Character specifying the type of blank correction that should be used. Options: "One milliQ" (default) or "Moving average". See Details for requirements for each type.

Details

The function runs following steps for individual data sheets:

1. Read in data

   Reads data from a MS Excel file using the openxlsx package. The Excel sheets have to be formatted in a specific way explained in Figure 1. Each data sheet (i.e. tab) is allowed to contain only three types of columns: one wavelength column, blank columns and sample columns. In other words, meta-data should not be included in the sheet, but these can be merged with data afterwards using the compile_data function.

   • Wavelength column Only one is allowed, and this column should contain the word "wavelength" as column name. Case does not matter and the column name is allowed to contain other letters such as unit. Wave lengths should be organized row-wise.

   • Blank columns (milliQ samples) should contain the letters "MQ" in capital letters, but are allowed to contain other characters too. Any remaining columns that do not contain the letters "MQ" or "wavelength" are assumed as sample columns.

   • Sample columns should contain either the unique sample code (CDO-001 for example) or three numbers containing the unique numeric label (001 for example) as a part of their column name. These codes are used to merge CDOM data with meta-data in compile_data function.

   

   Figure 1. Example how CDOM Excel sheets should be arranged. Note that commas (,) should not be used as decimal separators as in this example. Configure Excel to use periods (.) as decimal separators instead.

2. Blank correction

   Two methods for blank correction (i.e. subtraction of milliQ values) are implemented:

   • "One milliQ" One MQ is subtracted from all spectra in an Excel sheet. If this method is selected, only one MQ column per sheet is allowed. The order of sample columns does not matter.

   • "Moving average" Blank correction is done using a classic instrument drift correction routine by calculating moving averages of milliQ samples on both sides of samples and subtracting these moving averages from sample values. If this method is selected, sample and MQ columns have to be ordered in the same order they were analyzed.

3. Baseline correction

   Baseline correction is done by subtracting the average absorption in the range 600-650 nm from each spectra. This routine introduces the assumption that absorbance is equal to zero in the range 600-650 nm attributing any variation from zero to instrument baseline drift, temperature, scattering, and refractive effects (see Spectral corrections and S determination section in Material and Methods in Helms et al. 2008 and Green & Blough 1994).

4. Conversion from absorbance to absorption coefficients

   Absorbances are converted to absorption coefficients using the following equation from Helms et al. (2008):

   2.303 * A(λ) / l

   Where A(λ) is the absorbance for wavelength λ and l the path length in meters. 0.1 was used as path length.

5. Calculate spectral slopes for 275-295 and 350-400 nm using linear model fitting

   See Value for details about spectral slope calculation

6. Calculate spectral slopes for 300-650 nm using nonlinear model fitting.

   See Value for details about spectral slope calculation.

7. Return a list of variables

   A list of class CDOMdata is returned. See Value for details.

The routine above is conducted by a hidden function named .cdom_data. The process_cdom_data function loops the .cdom_data function over all tabs in an Excel file if sheet = "all" or over selected tabs, if a vector containing sheet numbers or names is supplied. Another hidden function .combine_cdom_data combines data provided by .cdom_data and generates a progress bar as running the routine over large files might take some time.

Value

Returns a list of class CDOMdata containing a data frame of spectral absorption coefficients ($spectra) and a data frame of desired variables ($data) to be exported to the database. The element data contains following columns:

• sample_name The standardized sample name that can be linked to meta-data

• abs254 Absorption coefficient (m-1) for 254 nm wavelength

• abs350 Absorption coefficient (m-1) for 350 nm wavelength

• abs375 Absorption coefficient (m-1) for 375 nm wavelength

• abs440 Absorption coefficient (m-1) for 440 nm wavelength

• sl295 Spectral slope for wavelengths between 275 and 295 nm. The slope is calculated using linear approximation after Helms et al. (2008). Absorption coefficients are logarithm transformed prior fitting using a linear model (see lm). The values are reported as positive exponential numbers following the mathematical convention of fitting to an exponential decay (see sl650), but are actually negative. The slopes are expressed in µm-1. If the slope is NA, some of the absorption coefficients were negative over the spectral range (275-295 nm), making it impossible to logarithm transform the values.

• sl400 Spectral slope (µm-1) for wavelengths between 350 and 400 nm. See sl295 for details.

• slope_ratio Slope ratio between sl295 and sl400 (sl295/sl400). The ratio is calculated using the exponential slopes.

• sl650 Spectral slope (µm-1) for wavelengths between 300 and 650 nm. The slope is calculated by fitting a nonlinear exponential model on absorpation coefficients using the nls function. The equation is from Helms et al. (2008) and can be expressed as follows:

  a(λ) = a(0) * exp(-sl650 * wavelength)

  where a(λ) are the absorption coefficient values over the spectrum 300 - 650 nm, a(0) the intercept and sl650 the slope. The nonlinear estimation can handle negative absorption coefficient values, unlike the linear approximation, and produces very similar results to linear approximation (Helms et al. 2008). The downside with nonlinear estimation is that it might not manage to fit the function on data that contain heavy instrumental errors. Currently the function does not include a buffer against this and may return an error, when used on bad data.

Author(s)

Mikko Vihtakari, Alexey Pavlov

References

Green, S.A., Blough, N. V, 1994. Optical absorption and fluorescence properties of chromophoric dissolved organic matter in natural waters. Limnol. Oceanogr. 39, 1903–1916. doi:10.4319/lo.1994.39.8.1903

Helms, J.R., Stubbins, A., Ritchie, J.D., Minor, E.C., Kieber, D.J., Mopper, K., 2008. Absorption spectral slopes and slope ratios as indicators of molecular weight, source, and photobleaching of chromophoric dissolved organic matter. Limnol. Oceanogr. 53, 955–969. doi:10.4319/lo.2008.53.3.0955

Stedmon, C.A., Markager, S., Kaas, H., 2000. Optical Properties and Signatures of Chromophoric Dissolved Organic Matter (CDOM) in Danish Coastal Waters. Estuar. Coast. Shelf Sci. 51, 267–278. doi:10.1006/ecss.2000.0645

See Also

plot.CDOMdata and print.CDOMdata for methods to use with CDOMdata objects.

compile_data to combine CDOM data with meta-data.

MikkoVihtakari/MarineDatabase documentation built on July 7, 2020, 2:16 a.m.