In edwardpmorris/FASTSpectra: Parse and radiometrically correct Ocean Optics spectra files

This is an example of the data processing of ground-based Ocean Optics USB2000 (OO-USB2000) spectroradiometer measurements.

Calculations have been documented in an R package to improve reproducibility.

Fist we need to load the R package, as it is not on CRAN it needs to be downloaded from GitHub, this can be done directly with the package devtools.

#install.packages("devtools")

#library(devtools)

#install_github("edwardpmorris/FASTSpectra")

library(FASTSpectra)

Details about each of the functions are now available via R help, for example by typing help("FASTspectra") in the console. FASTSpectra essentially uses hyperSpec as a framework for parsing, calibrating and other performing other operations on spectral data.

A working example

A number of ground-based, paired 'sample' and 'reference' spectra of different ground types (concrete path, green-lawn and brown-lawn) were collected during an instrument inter-comparison conducted at Field Spectrometry Facility (FSF), Edinburugh.

The spectra collected with the OO-USB2000 are installed with the package, to use them we need to set the file path.

file.path <- system.file("extdata", package = "FASTSpectra")

Then we can parse the spectra into hyperSpec object using the function specific for reading OO text files.

dn <- FASTSpectra::read.txt.OceanOptics(files=paste0(file.path,"/*.txt"))

Details about the hyperSpec object can be accessed by typing the object name in the console, dn. If you do this you will see there are 44 spectra with a wavelength range between 339.47 to 1028.84 nm and 2048 data points per spectra. Metadata associated with each spectra is also included as a dataframe with 15 columns, including information such as the filename, integration-time and timestamp of the measurement.

We can also access all of the hyperSpec functions, especially designed for working with spectral data, for example plotting.

library(hyperSpec)

hyperSpec::plotspc(dn)

Looking at the plot we can see a mix of spectra with high and low values in the mid wavelengths (~550 nm), that is because the 'sample' (S) and 'reference' (R) measurements are all in one file.

Which is which is defined in a 'field log', a .csv file with columns id, S, and R, we can assign a column in the metadata using the FASTSpectra::assign.type function:

# assign measurement id and type using fieldlog file
logfile <- paste0(file.path,"/fieldlog.csv")

dn <- FASTSpectra::assign.type(dn, logfile=logfile)

Now if we check the metadata (type dn in console) we will see 2 new columns, type and id. These can be used to split the hyperSpec object into sample and reference spectra.

# split into sample and reference spectra
dn <- hyperSpec::split(x=dn, f=dn@data$type)

Radiometric calibration

Spectra, as instrument values, called digital number (DN) counts, require conversion to radiometric units using instrument specific metadata and calibration values (instrument response functions). For this we use the FASTSpectra::rad.cor function, which allows the calculation of normalised DN, radiant energy, spectral flux, spectral intensity, spectral radiance and spectral irradiance.

These only make sense if the 'measurement geometry', 'integration time' [s] of a measurement, 'surface area of the instrument light apature' [m], and 'instrument-specific response function' in radiant energy (e.g., J/count or other units) is known and supplied to the function.

Luckily, with help from FSF with have all of these, and can go ahead and calculate the spectral irradiance [W / m2 nm] of the sample spectra.

# set path of instrument-specific response function
cal.rad <- paste0(file.path,"/*.IrradCal")

cal <- import.calibration(cal.rad)

plot(cal)

# convert [DN/nm] to [W / m2 nm]
sam <- rad.corr(dn$SAMP,
                type="spectral.radiance",
                cal.DN2RadiantEnergy = cal.rad
                )

plot(sam)

Reference panel

We can also do the same for the reference measurements, however for these we also need to include the reference panel calibration i.e., the reflectance of the reference panel relative to a 'lambertian' surface with a reflectance factor of 1 throughout the wavelength range of interest.

# convert [DN/nm] to [W / m2 nm] relative to lambertian reference panel 
cal.ref <- paste0(file.path,"/*.ReflCal")

ref <- rad.corr(dn$REF,
                type="spectral.irradiance",
                is.REF=TRUE,
                cal.DN2RadiantEnergy = cal.rad,
                cal.RRefPanel = cal.ref
                )

We can no plot each set of spectra and observe that indeed the reference measurements are generally much higher than the samples.

plot(sam, wl.range=380:950)

plot(ref, wl.range=380:950)

Reflectance

Now that our spectra are calibrated we can go ahead and calculate the 'absolute' reflectance for each specta pair (defined by the logfile).

# calculate reflectance as 'irradiance/irradiance'
SHR <- calc.reflectance(reference=ref, sample=sam, logfile=logfile)

# plot absolute reflectance
plot(SHR, wl.range=380:950)

In this plot of absolute or 'hemispherical-conical' reflectance the groupings of the spectra from measurements at the same positions on the 3 ground types becomes a bit clearer. Around 7 spectra were collected in a 'round-robin' fashion, i.e., one ground type and then the next, in variable lighting conditions (it was summer in Scotland).

To get a feeling for the variability in the measured reflectance spectra we can group the spectra by ground type and calculate quartiles.

# calculate quartiles by ground type
SHR.quant <-
  aggregate(
  SHR,
  SHR@data$id,
  quantile,
  probs = c(0.05, 0.16, 0.5, 0.84, 0.95),
  na.rm = T
  )

plot(SHR.quant
  , col = matlab.dark.palette(length(unique(
  SHR.quant@data$.aggregate
  )))
  , fill=SHR.quant@data$.aggregate
  , wl.range = 380:950)

Radiometric calculations in detail

A diagram representing the workflow and the components required to derive radiometric quantities from ground-based spectrometry measurements (using a OO-USB2000) is shown below.

require(DiagrammeR)
require(DiagrammeRsvg)
svg <- export_svg(grViz("workflow.DOT", width = 600, height = 400))
write(svg, "workflow.svg")
system("inkscape -z -e workflow.png -w 600 -h 400 workflow.svg")

Green nodes represent 'constant' metadata about the instrument and reference panel, for example the 'instrument response function' and the diameter of the collection apature. Blue nodes represent metadata about a specific measurement that may potentially change from one spectra to the next.

Grey nodes represent spectral 'units', with Digital Number counts (DN) being instrument values and the usual starting point for radiometric conversion. Here it is assumed that corrections for non-linearity and dark signal are done previously (for example by the spectrometer software).

Terminology and symbols for radiometric units and metadata follow ISO standards[REF] were possible; some extra terminology in use within the discipline are also included (Table 1).

TABLE 1 Name, Symbol, Unit, Description, Link

Instrument-specific properties

Conversion of instrument Digital Number counts to radiometric units requires instrument- and measurement-specific metadata including;

• The 'integration time' (t, s) of a measurement i.e., the time period over which the instrument records light through the measuring apature.

• The viewing geometry of measurements ($\theta$~v~, °), expressed as an angle from the nadir (0°) viewing zenith. In general most ground-based spectrometry measurements aim for a $\theta$~v~ of 0°.

• The 'spectral sampling interval' ($\lambda$, nm) i.e., the wavelength scale.

• The 'diameter' (d, m) and 'full angle' or 'angular field of view' ($\theta$~fov~, °) of the instrument light collection apature. For the OO-USB2000 these are often properties of the fibre optic that guides light into the detector array. Full angle can be calculated from the 'numerical apature' of the optical system.

• The 'instrument-specific response function' or 'spectral response of each band' (IRF~$\lambda$~, J count^-1^). This may have different units, but must cover the full range of wavelengths of interest and should have the same spectral resolution as the instrument.

• For reflectance measurements using a reference panel, the spectral hemispherical reflectance of the panel, covering the full range of wavelengths of interest.

From these properties the 'solid angle of the collector' in steradians (Ω, sr) is calculated as:

$${\Omega} = \pi \times \sin \left( \frac{\theta_{fov}}{2} \times \frac{\pi}{180} \right)^{2}$$

The surface area of the collection apature (A, m^2^) as:

$$A = \pi \times \left(\frac{d}{2}\right)^2$$

The projected area of the collection apature (Acos$\theta$, m^2^) as:

$${A \cos \theta} = A \times \cos (\theta_{v} \times \frac{\pi}{180})$$

'Wavelength spread' or 'half-power bandwidths' (d~$\lambda$~, nm), i.e., the half-width of each band is calculated as:

$$d_{\lambda} = \frac{\lambda_{i} - \lambda_{i+1}}{2}$$

Normalised Digital Number counts (DN~norm~) [counts s^-1^ nm^-1^]

Calculations begin with conversion of instrument counts spectra (DN~$\lambda$~) to normalised DN (DN~norm,$\lambda$~, counts s^-1^ nm^-1^) using the 'integration time' (t, s) i.e., the time period over which the instrument records light through the measuring apature, and the 'wavelength spread' i.e., the width of each band (d~$\lambda$~, nm).

$${DN}{\mathrm{norm, \lambda}} = \frac{{DN}{\lambda}}{t \times d_{\lambda}}$$

Spectral flux (Φ~e,λ~) [W / nm]

Using the instrument-specific response function (IRF~$\lambda$~, J count^-1^), spectral flux is calculated as:

$$\Phi_{\mathrm{e, \lambda}} = {{DN}{\mathrm{norm, \lambda}}}\times{{IRF}{\lambda}}$$

Spectral intensity (I~e,Ω,λ~) [W sr^-1^ nm^-1^]

Using the solid angle of the collector apature in steradians (Ω, sr) spectral intensity can be calculated as:

$$I_{\mathrm{e},\Omega, \lambda} = \frac{ \Phi_\mathrm{e, \lambda}}{ \Omega}$$

Spectral radiance (L~e,Ω,λ~) [W sr^-1^ m^-2^ nm^-1^]

Using the projected area of the collector apature (Acos$\theta$), spectral radiance is calculated as:

$$L_{\mathrm{e},\Omega, \lambda} = \frac{ I_{\mathrm{e},\Omega, \lambda}}{ A \cos \theta}$$

Spectral irradiance (E~e,λ~) [W m^-2^ nm^-1^]

Using the surface area of the collection apature (A, m^2^), spectral irradiance is calculated as:

$$E_\mathrm{e, \lambda} = \frac{\Phi_\mathrm{e, \lambda}}{A}$$

Spectral hemispherical reflectance (R~λ~)

Assuming both the spectral radiant flux reflected and recieved by the surface (Φ^r^~e,λ~ and Φ^i^~e,λ~) are not dependent on direction and are measured with an instrument that has the same collection area, hemispherical reflectance is defined as:

$${R_{\lambda}} = \frac{\Phi_{\mathrm{e},\lambda}^{\mathrm{r}}}{\Phi_{\mathrm{e},\lambda}^{\mathrm{i}}}$$

In practice this assumption is valid for the incident flux reflected from a reference panel or up and downward fluxes measured using a cosine collector and the same instrument.

Often spectral irradiance reflectance is more conveniently calculated as:

$${R_{\lambda}} = \frac{E_{\mathrm{e},\lambda}^{\mathrm{r}}}{E_{\mathrm{e},\lambda}^{\mathrm{i}}}$$

Spectral directional reflectance (R~λ,Ω~)

When assuming directionality in both upwelling and downwelling light, then spectral radiance reflected and recieved by the surface (L^r^~e,Ω,λ~ and L^i^~e,Ω,λ~) are used to calculate reflectance as:

$$R_{\Omega,\lambda} = \frac{L_{\mathrm{e},\Omega,\lambda}^\mathrm{r}}{L_{\mathrm{e},\Omega,\lambda}^\mathrm{i}}$$

Spectral directional hemispherical reflectance (R^h^~Ω,λ~) [sr^-1^]

In practice the optical geometry of most ground-based spectrometry measurements (and satelite sensors) is a mix between hemispherical and directional measures. Sensors usually have limited anglular field of view and incident radiance is reflected equally in all directions by the lambertian reference panel.

Specifically, as defined for remote-sensing reflectance [REF];

...a measure of how much of the downwelling radiance that is incident on a surface in any direction is eventually returned through the surface into a small solid angle $\Delta \Omega$ centered on a particular direction $(\theta,\phi)$.

Assuming the spectral radiance reflected by the target surface and recieved by the collector surface is directional (L^r^~e,Ω,λ~) and the incident radiant flux reflected by a reference panel or collected by a cosine sensor is hemispherical (E^i^~e,λ~), directional hemispherical reflectance is defined as:

$${R_{\Omega,\lambda}^{\mathrm{h}}} = \frac{L_{\mathrm{e},\lambda}^{\mathrm{r}}}{E_{\mathrm{e},\lambda}^{\mathrm{i}}}$$

edwardpmorris/FASTSpectra documentation built on May 15, 2019, 11:03 p.m.