In Echisholm21/vprr: Processing and Visualization of Video Plankton Recorder Data

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = FALSE
)

library(vprr)

Section 1: Background

This document was produced at Bedford Institute of Oceanography (BIO) to accompany the vprr package, a processing and visualization package for data obtained from the Digital Auto Video Plankton Recorder (VPR) produced by SeaScan Inc. The VPR consists of a CPU, CTD, and camera system with different optical settings (i.e., magnifications). It captures underwater images and records their corresponding salinity, temperature, and depth. The vprr package functions to join environmental and plankton data derived from the CTD and camera, respectively, and calculate plankton concentration and averaged environmental variables along the path of the VPR. The package does not include automated image classification; however, there is an optional manual classification module, which can be used to review and correct outputs from automated image classification while providing a record of any (re)classifications.

The VPR outputs two raw files (.dat and .idx) for a given time period in a deployment. These files are processed together in a software provided with the VPR (i.e., AutoDeck), which decompresses the images, extracts "regions of interest" (ROIs), and outputs ROI image files and a corresponding CTD data file (.dat). The ROI file names are numeric consisting of 10 digits. The first 8 digits correspond to the number of milliseconds elapsed in the day at the time the image was captured. The last two digits correspond to the ROI identifier (01-99). The ROIs and corresponding CTD data are linked by their 8 digit time stamp. After the ROIs have been extracted from the raw files they may be sorted into categories manually or by an automated classification procedure. In vprr, file naming conventions and directory structures are inherited from a VPR image classification and analysis software, Visual Plankton. However, the functionality of vprr is not dependent on the use of Visual Plankton.

The data inputs for processing in vprr consist of the following file types: aid (.txt), aidmeas (.txt), and CTD (.dat). The aid and aidmeas files are derived from separate image classification and measurement steps outside of vprr. Each "aid" (i.e., autoid) file contains file paths to individual ROIs that have been classified to the category of interest. The corresponding "aidmeas" file contains morphological data for the ROIs (e.g., long axis length, perimeter, etc.).

{width=100%}

Section 2: Summary of vprr data processing steps

The processing steps performed in vprr are detailed in Figure 2, along with the associated data outputs.

{width=100%}

Processing Environment

Before beginning data processing with vprr, it is recommended that a processing environment be created containing commonly used variables and file paths. The simplest and most reproducible way to achieve this is to write an R script where all the mission and system specific variables are contained, then save the environment as a RData file to be loaded at the start of any processing scripts. This processing environment contains reference to a station identifier csv file which should be created for each mission. This file links station names from deck sheets to the day and hour values assigned by AutoDeck. Day and hour values represent the Julian day (3 digit) and two digit hour (24 hour clock) when sampling was done. Note that the day and hour values will be in the time zone of the computer used to run AutoDeck. Ensure that this matches the time zone of the VPR CPU at the time of data collection to avoid a time offset between data sources.

Another important part of setting up the processing environment is ensuring the proper directory structure is in place, see Appendix 1 for details on the required directory structure.

#### set VPR processing environment --------------------------------------

# setwd('~/COR2019002/')
day <- c('222', '222') # 3 digits
hr <- c('03', '04') # 2 digits
cast <- '5' # chr
station <- 'Example' # chr
cruise <- 'COR2019002' # chr
opticalSetting <- 'S2' # chr
dh <- paste0('d', day, '.h', hr)
station_of_interest <- paste0('vpr', cast, '_', station)
binSize <- 3 # num (metres)
imageVolume <- 41439 # num
year <- '2019' # chr

# location for new autoid folder
new_autoid <- 'new_autoid/'
# location of original ROIs
roi_path <- 'extdata/'
# location of CNN aid files
manual_class_basepath <- 'extdata/COR2019002/autoid'
# location of manual classification records
manual_record_path <- 'extdata/COR2019002/manual_reclassification_record/'
# location of CTD data
castdir <- 'extdata/COR2019002/rois/vpr5/d222'
# get categories
categories <- list.files(new_autoid, include.dirs = TRUE)

# Save processing environment
# save.image(file = paste0(cruise, "env.RData"))

Once this environment is set, it can be loaded into any processing session by using

load('COR2019002_env.RData') # where COR2019002 is cruise name

If sharing processing code with colleagues on version control, keeping the environment variables separate (outside of the git project) will allow collaboration while avoiding inconsistencies in file paths or folder names.

Image Copying (optional):

ROIs are organized into folders corresponding to their assigned classification categories from automated image classification. The information in each aid file is used to create a folder of images that have been classified to that category. This step is only required if manual re-classification (see Section 2.2) is intended. Further details on image copying are provided in Section 3.

Manual re-classification (optional):

Automated classifications from are manually checked, which allows for manual correction and addition of categories not previously used for automated classification. ROIs that have been copied are manually sorted to correct for misclassifications. Updated aid and aidmeas files are produced. Further details on manual re-classification are provided in Section 4.

Data Processing:

Data outputs including CTD (.dat files), automated classifications (aid files) and measurements (aidmeas files) are joined together. The aid and aidmeas files, which may have been updated, are joined with CTD text files by the 8 digit time stamp (ROI number). The data are then averaged in user-defined vertical bins to produce a time series of plankton concentrations and environmental variables. Quality controlled data products (before and after binning) are then exported in simple formats (csv, RData, oce) for plotting and analysis. Further details on data processing are provided in Section 5.

Section 3: Image Copying

In this step, ROIs are copied to folders that are organized based on the day and hour of data collection and classification category assigned from automated classification (see Appendix 1: 'Image Folders'). The images are organized by AutoDeck into day and hour; however, reorganizing them based on classification allows easier human interaction with the data and visual inspection of classifications. Moreover, this directory structure is used by the next step of processing (i.e., manual re-classification). To implement this step use the function vprr::vpr_autoid_copy() For more information on input variables, please see documentation for vpr_autoid_copy()

#### run image organization -----------------------------------------------
vpr_autoid_copy(new_autoid = new_autoid,
            roi_path = roi_path,
            day = day,
            hour = hr,
            cast = cast,
            station = station,
            org = 'dayhour'
            )

Section 4: Manual Re-classification

Manual re-classification of some categories after automated classification may be required to achieve identification accuracy standards. In this step, ROIs are displayed on the screen one at a time for manual verification. If an image has been misclassified or if it falls into a new user-defined category (described below), the image can be re-classified. This is especially useful for classification of rare categories that were not defined prior to automated classification. After completing manual re-classification for a day-hour set, new aid and aidmeas files are created for new categories, which are identical in format to original aid and aidmeas files.

Section 4.1: Preparing the environment by setting some variables

• Load the processing environment, which includes the basepath variable.
• Set day and hour of interest.
• (optional) Set category of interest. These categories are the existing automated classification categories which require manual re-classification, as well as any new categories. The vprr::vpr_category_create() function sets up the folder structure for any new categories which have been added to the list of interest.
• Note that if you create new categories part way through manual classification, no misclassified files will have been created in this category for hours which you have already processed. This may cause some unexpected errors down the line in processing, you could either reprocess previous hours of data with all the new categories or manually create empty misclassified files.
• Run manual re-classification with vprr::vpr_manual_classification(). This function has a few optional arguments to customize the manual re-classification experience, notably gr which is a logical value determining whether or not manual re-classification options appear as pop ups or in the console, as well as img_bright, a logical which determines whether or not the original image is appended with an extra bright version of the image. Having a bright version of the image allows the user to see the outline of the organism better, any thin appendages become more clear and gelatinous organisms like chaetognaths or ctenophores are easier to distinguish.
• The threshold argument in vpr_manual_classification() can be used to determine the minimum automated classification confidence required for images to be passed through without manual verification.

  #### MANUAL RE-CLASSIFICATION -------------------------------------
  # Once automated classification is complete
  # verify classification accuracy by manually 
  # looking through select images
  
  load('COR2019002_env.RData')
  
  # category_new <-
  #   c(
  #     "veliger_gastropod"
  #     )
  #
  # add new category
  # vprr::vpr_category_create(taxa = category_new, basepath = auto_id_folder)
  
  vpr_manual_classification(day = day,
                    hour = hr,
                    basepath = manual_class_basepath,
                    gr = FALSE)
  

Section 4.2: Generate new aid and aidmeas files

The function vprr::vpr_manual_classification() produces two files ('misclassified' and 're-classified' text files) as a record of manual re-classification, which are found in the R project working directory in folders named by the day and hour that the data were collected. The function vprr::vpr_autoid_create() takes these files and outputs new aid and aidmeas files in the R working directory in folders named by classification category. This step should be run after each hour of data is manually re-classified. If aidmeas files have not been created (through a separate measurement workflow), these functions will run on just the aid files without issue.

#### REORGANIZE ROI AND ROIMEAS DATA ---------------------------------------
# get mis/re classified files
manual_record <- list.files(file.path(manual_record_path, dh), 
        full.names = TRUE)
misclassified <- grep(manual_record, 
      pattern = "misclassified",
      value = TRUE)
reclassify <- grep(manual_record, pattern = "reclassify", value = TRUE)

# MOVE ROIS THAT WERE MISCLASSIFIED INTO CORRECT FILES & REMOVE MISCLASSIFIED ROIS
vpr_autoid_create(reclassify, misclassified, manual_class_basepath,
      mea = FALSE, categories = categories)

The aid and aidmeas files are both text files which are specifically formatted to record classification outputs for further processing. The format and naming conventions of these files has been inherited from a VPR image classification and data processing tool called Visual Plankton (written in Matlab); however, the functionality of vprr is independent from that of Visual Plankton. The aid files are text records of image paths, where each individual text file represents a classification category. Each line of the aid file is the full path to an image which was classified into the designated category. Note that the naming scheme of aid files does not include the category name in the file title and the category is only identifiable by the folder in which it is located. For example the 'krill' classification aid file might be named 'oct10_1svmaid.d224.h01' but be located within the 'krill' autoid folder. The aidmeas files are also text files which represent a variety of different measurements taken of the object(s) within a ROI image. The columns of the aidmeas files are c('Perimeter', 'Area', 'width1', 'width2', 'width3', 'short_axis_length', 'long_axis_length'). The aidmeas files were originally created during processing with Visual Plankton but are not created or required for processing with vprr. We have begun development of an ImageJ workflow which will create an equivalent product.

Examples of each of these files can be found below.

aid <- read.table(file = system.file("extdata/COR2019002/autoid/bad_image_blurry/aid/sep20_2svmaid.d222.h04",
                                     package = 'vprr', mustWork = TRUE))

head(aid)

aidmeas <- readLines(
  system.file("extdata/COR2019002/autoid/bad_image_blurry/aidmea/sep20_2svmaid.mea.d222.h04",
                                 package = 'vprr', mustWork = TRUE))

head(aidmeas)

Section 4.3: File check

The last step of manual re-classification includes some manual file organization and final checks. These files should be manually reorganized in a new directory which will become the new auto_id_folder (see Appendix 1: Directory Structure). Any aid and aidmeas files from categories which were not manually checked and re-classified should also be added to this new auto_id_folder if they are to be included in further processing (e.g., computation of concentration in user-specified depth bins). After the updated aid and aidmeas files have been manually reorganized they can be quality controlled using vprr::vpr_autoid_check(). The user could also manually check the files.

#### FILE CHECK ----------------------------------------------------------
# (outputs text file in working directory with check data)
vpr_autoid_check(new_autoid, original_autoid = manual_class_basepath, 
             cruise = cruise, dayhours = dh)

Section 5: Data Processing

This is the main chunk of coding required to generate data products. This step does not require image copying (Section 3) or manual re-classification (Section 4) steps; however, if these steps were taken the aid and aidmeas files generated from manual re-classification and integrated into the directory structure (as specified in Section 4) are used as an input. The following is a walk-through of processing data from a DFO field mission (i.e. mission COR2019002) in the southern Gulf of St. Lawrence in 2019. First, all libraries should be loaded and the processing environment, described in Section 2.4 should be loaded.

##### PROCESSING  ---------------------------------------------------------
library(vprr)

#### FILE PATHS & SETTINGS ------------------------------------------------
# loads processing environment specific to user

load('COR2019002_env.RData')

CTD data are loaded in using vprr::vpr_ctd_read. During CTD data read in, a seawater density variable sigmaT is derived using the function oce::swSigmaT, and depth (in meters) is derived from pressure using the function oce::swDepth. For more information on the oce package, see dankelley/oce on GitHub.

#### READ CTD DATA --------------------------------------------------------
ctd_files <- list.files(castdir, pattern = '.dat', full.names = TRUE)
ctd_dat_combine <- vpr_ctd_read(
  ctd_files,
  station_of_interest,
  col_list = c(
    "time_ms",
    "conductivity",
    "temperature",
    "pressure",
    "salinity",
    "reserved_mv",
    "turbidity_mv",
    "fluorescence_mv",
    "oxygen_mV",
    "pitch_degrees",
    "roll_degrees",
    "frame_number"
  )
)

The aid (and aidmeas) files, which reflect manual classification (if used, see Section 4), are then found

##### FIND VPR DATA FILES ---------------------------------------------
# find aid files 
auto_id_path <- list.files(new_autoid, full.names = TRUE)
# Path to aid for each category
aid_path <- paste0(auto_id_path, '/aid/')

# AUTO ID FILES
aid_file_list <- list()
for (i in 1:length(dh)) {
  aid_file_list[[i]] <-
    list.files(aid_path, pattern = dh[[i]], full.names = TRUE)
}

aid_file_list_all <- unlist(aid_file_list)
remove(aid_file_list, aid_path) # tidy up environment

# Path to aidmeas for each category
aidmeas_path <- paste0(auto_id_path, '/aidmea/')

# AUTO ID (MEASUREMENT) FILES
aidmea_file_list <- list()
for (i in 1:length(dh)) {
  aidmea_file_list[[i]] <-
    list.files(aidmeas_path, pattern = dh[[i]], full.names = TRUE)
}

aidmea_file_list_all <- unlist(aidmea_file_list)
remove(aidmea_file_list, aidmeas_path) # tidy up environment

aid (and aidmeas) files are read in using vprr::vpr_autoid_read().

  ##### READ ROI DATA -----------------------------------------------------
  roi_dat_combine <- vpr_autoid_read(
      file_list_aid = aid_file_list_all,
      export = 'aid',
      station_of_interest = station_of_interest,
      opticalSetting = opticalSetting,
      categories = categories
  )

  #### READ MEASURMENT DATA -----------------------------------------------
  meas_dat_combine <- vpr_autoid_read(
      file_list_aid = aid_file_list_all,
      file_list_aidmeas = aidmea_file_list_all,
      export = 'aidmeas',
      station_of_interest = station_of_interest,
      opticalSetting = opticalSetting,
      categories = categories
  )

Next, CTD and aid data are merged to create a data frame describing both environmental variables (eg. temperature, salinity) and classified images. The function used is vprr::vpr_ctdroi_merge().

##### MERGE CTD AND ROI DATA ------------------------------------------
ctd_roi_merge <- vpr_ctdroi_merge(ctd_dat_combine, roi_dat_combine)

Before final export of data products, the following variables are added to the data frame: time in hours (time_hr) is calculated, and a time stamp (ymdhms) with POSIXct signature in Y-M-D h:m:s format is added using the function vpr_ctd_ymd.

##### CALCULATED VARS -------------------------------------------------
# add time_hr and sigma T data and depth
data <- ctd_roi_merge %>%
    dplyr::mutate(., time_hr = time_ms / 3.6e+06)
data <- vpr_ctd_ymd(data, year)

Average plankton concentration and environmental variables (e.g., temperature, salinity, density, etc.) are then computed within a user defined depth bin. The computation of plankton concentration is dependent on the assumption that the same animals are not re-sampled by the instrument. The bin-averaging step standardizes plankton concentrations when the VPR does not sample the water column evenly. This can occur due to characteristics of the deployment or variability in the sampling rate, which is not necessarily constant in older versions of the VPR. Binning also reduces noise in the data. First, an oce CTD object is created using vprr::vpr_oce_create(). Then, bin-averaging is done using vprr::bin_cast(). Concentrations are calculated for each category of interest.

##### BIN DATA AND DERIVE CONCENTRATION ---------------------------------
# create oce object
ctd_roi_oce <- vpr_oce_create(data)

# bin and calculate concentration for all category (combined)
vpr_depth_bin <- bin_cast(ctd_roi_oce = ctd_roi_oce,
    binSize =  binSize,
    imageVolume = imageVolume)

# bin and calculate concentrations for each category
category_conc_n <- vpr_roi_concentration(data,
    categories,
    station_of_interest,
    binSize,
    imageVolume)

Measurement data can also be binned to match the resolution of ROI data.

#### BIN SIZE DATA -------------------------------------------------------

size_df_f <- vpr_ctdroisize_merge(data,
    data_mea = meas_dat_combine,
    category_of_interest = c("krill"))

size_df_b <- vpr_size_bin(size_df_f, bin_mea = 3)

Finally, data are saved as RData and csv files for export and plotting. Data are also saved as an oce object in order to preserve both data and metadata in an efficient format. This example shows gathering metadata from a csv but it could also be input manually.

##### SAVE DATA --------------------------------------------------------
#Metadata (from csv)
vpr_summary <- read.csv('vpr_metadata_COR2019002.csv')
vpr_summary_st <- vpr_summary[vpr_summary$event == as.numeric(cast), ]
startlat <- unique(vpr_summary_st$latitudeStart)
stoplat <- unique(vpr_summary_st$latitudeStop)
startlon <- unique(vpr_summary_st$longitudeStart)
stoplon <- unique(vpr_summary_st$longitudeStop)
zstn <- unique(vpr_summary_st$soundingStart)
ddate <- unique(vpr_summary_st$dateStart)
tstart <- unique(vpr_summary_st$timeStart)
tend <- unique(vpr_summary_st$timeStop)
vpr_comment <- unique(vpr_summary_st$comment)

#Save file
oce_dat <- vpr_save(category_conc_n,
      metadata = list('deploymentType' = 'vprProfile',
      'waterDepth' = zstn,
      'serialNumber' = 'DAVPR-00',
      'latitudeStart' = startlat,
      'latitudeStop' = stoplat,
      'longitudeStart' = startlon,
      'longitudeStop' = stoplon,
      'castDate' = ddate,
      'castStartTime' = tstart,
      'castEndTime' = tend,
      'processedBy' = 'J. Doe',
      'opticalSetting' = opticalSetting,
      'imageVolume' = imageVolume,
      'comment' = vpr_comment))

# oce data and metadata object
save(file = paste0(savedir, '/oceData_', station,'.RData'),
    oce_dat)

# Save RData files
# VPR and CTD data
save(file = paste0(savedir, '/stationData_', station,'.RData'),
    data)

# binned data including measurements
save(file = paste0(savedir, '/bin_size_dat_', station_of_interest,'.RData'),
    size_df_b)

# Write csv files
# VPR and CTD data with concentrations by taxa
write.csv(file = paste0(stdir, '/vpr_data_binned', station, '.csv'),
    category_conc_n)

# measurement data
write.csv(file = paste0(stdir, '/vpr_meas', station_of_interest, '.csv'),
    roimeas_dat_combine)

Section 6: Disclaimer

The functions in vprr were created for a specific project and have not been tested on a broad range of field mission data. It is possible that deviations in data format and directory structure from that described herein may result in errors when using vprr. The vprr package was developed for the purpose of processing data collected during tow-yo VPR deployments and image classification. The purpose of this document is to provide a template for processing and visualizing VPR data that can be adapted by other users for their own objectives.

Appendix 1: Directory Structure

The directory structure required is described below

• C:/

  • data

    • cruise_name

      • autoid

        • category
          • aid
          • aidmea
          • image folders

      • rois

        • vprtow#
          • day
            • hour

      • trrois

        • vprtow#
          • day
            • hour

This is your project directory, where your R scripts and work products will be stored:

• ...

  • VPR_PROJECT

  • R

    • R scripts/ workflows

  • new_autoid

    • category

    • aid

    • aidmea

  • manual_reclassification_record

    • day/hour

    • misclassified

    • reclassified

  • figures

  • station names (csv)

Appendix 2: Glossary

Aid files - Visual Plankton style file output text file, listing file path information for ROI's of a specific classification group

AidMeas files (AutoID measurements) - Visual Plankton style output text file, listing measurement data for ROI's of a specific classification group. Unit is pixels and columns are 'Perimeter', 'Area', 'width1', 'width2', 'width3', 'short_axis_length', 'long_axis_length'

Auto Deck - software which pulls plankton images from Video Plankton Recorder frames based on specific settings

Auto ID - The automated classification given to an image from a machine learning algorithm

AutoID files - Includes both Aid and AidMeas files as part of automated classifications

BIO - Bedford Institute of Oceanography, a research institute in Halifax NS, Canada

Classification category (category) - A defined group under which VPR images can be classified, often represents a taxonomic group (e.g. Krill), but can also be defined by image type (e.g. 'bad_image_blurry'), or other (e.g. 'marine_snow'), should be one continuous string (no spaces)

CTD - Conductivity, Temperature and depth sensor instrument

Day - Julian calendar day on which VPR data was collected (three digits)

Hour - Two digit hour (24 hour clock) describing time at which VPR data was collected

Image volume - The measured volume of water captured within a VPR image. Calculated based on optical setting and VPR standards. This is based on AutoDeck settings, it is calculated from the VPR calibration file (unique to each instrument). It will change based on AutoDeck settings and should be updated with each cruise/ processing batch. It is measured in cubic mm

Optical Setting - A VPR setting controlling image magnification and field of view, which can be S0, S1, S2 or S3, where S0 has the greatest magnification and smallest image volume, and S3 has the least magnification and largest image volume

ROI - Region of interest, images identified by autodeck within VPR frames based on settings defined in autoDeck program

SeaScan - Oceanographic instrument manufacturing company

station - A named geographic location, where the VPR was deployed

Tow-yo - A VPR deployment method where the VPR is towed behind a vessel while being raised and lowered through the water column in order to sample over both depth and distance

TRROIS - Training set of images used to train machine learning algorithm

VP - Visual Plankton program, written in Matlab to classify VPR images

VPR - Video Plankton Recorder, oceanographic instrument used to image small volumes of water for the purpose of capturing images of plankton

vprtow# - A numeric code which is unique to each VPR deployment

Working Directory - File path on your computer that defines the default location of any files you read into R, or save out of R

Echisholm21/vprr documentation built on Feb. 9, 2025, 4:21 p.m.