preprocess_metadata_tables: Preprocess metadata tables

In WorldFishCenter/peskas.timor.data.pipeline: Functions to Implement the Timor Small Scale Fisheries Data Pipeline

Preprocess metadata tables

Description

Takes ingested metadata tables, validates them, and processes them into artefacts (devices information, flags) to be used further down in the pipeline.

Usage

preprocess_metadata_tables(log_threshold = logger::DEBUG)

Details

Specifically this function:

• Validates the flags table containing the columns: flag_id, flag_category, flag_message

• Constructs a devices table containing the columns: imei

This function requires no arguments because it retrieves the parameters from conf.yml. The fields required are:

metadata:
  spreadsheet:
    name:
  version:
    preprocess:
storage:
  storage_name:
    key:
    options:
      project:
      bucket:
      service_account_key:

WorldFishCenter/peskas.timor.data.pipeline documentation built on April 14, 2025, 1:47 p.m.