In KopfLab/isoreader: Read Stable Isotope Data Files

library(isoreader)
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

This vignette introduces some of the development features of the isoreader package and is aimed primarily at code contributors interested in expanding its functionality or helping with bug fixes.

Adding new file format readers

Testing out new file format readers is easiest by registering a new reader function for a specific file extension using iso_register_dual_inlet_file_reader and iso_register_continuous_flow_file_reader, respectively. Both require an extension (e.g. ".ext"), name of the new reader function ("new_reader"), and optionally a description. Both functions automatically return a data frame with a list of all registered reader. Overwriting of existing readers with a different function requires an explicit overwrite = TRUE flag. All reader functions must accept an isoreader data structure object (ds) as the first argument, a list of reader specific options as the second argument (options), and should return the structure with data filled in for downstream isoreader operations to work smoothly. The following minimal example illustrates how to do this with the new_reader function simply printing out the layout of the provided data structure skeleton ds.

new_reader <- function(ds, options = list()) {
  isoreader:::log_message("this is the new reader!")
  str(ds)
  return(ds)
}

# register new reader
readers <- iso_register_dual_inlet_file_reader(".new.did", "new_reader")
knitr::kable(readers)

# copy an example file from the package with the new extension
iso_get_reader_example("dual_inlet_example.did") |> file.copy(to = "example.new.did")

# read the file
iso_read_dual_inlet("example.new.did", read_cache = FALSE)
file.remove("example.new.did")

Note that for parallel processing to work during the read process (parallel = TRUE), isoreader needs to know where to find the new reader function. It will figure this out automatically as long as the function name is unique but if this fails (or to be on the safe side), please specify e.g. env = "R_GlobalEnv" or env = "newpackage" during the reader registration. Also note that isoreader will not automatically know where to find all functions called from within the new reader function if they are not part of base R and it is recommended to make all outside calls explicit (e.g. dplyr::filter(...)) to preempt this potential problem. For info messages and warnings to work with the progress bar and in parallel reads, make sure to use isoreader:::log_message(...) and isoreader:::log_warning(...) instead of base R's message(...) and warning(...).

If you have designed and tested a new reader, please consider contributing it to the isoreader github repository via pull request.

Processing hooks

Isoreader defines two processing hooks at the beginning and end of reading an individual file. This is useful for integration into pipelines that require additional output (such as GUIs) but is also sometimes useful for debugging purposes. The expressions are evaluated in the context of the isoreader:::read_iso_file function and have access to all parameters passed to this function, such as e.g. file_n and path. Same as for new readers: for info messages and warnings to work with the progress bar and in parallel reads, make sure to use isoreader:::log_message(...) and isoreader:::log_warning(...) instead of base R's message(...) and warning(...). The main difference between the two is that log_message() will honor the quiet = TRUE flag passed to the main iso_read...() call whereas log_warning() will always show its message no matter the quiet setting.

isoreader:::set_read_file_event_expr({
  isoreader:::log_message(sprintf("starting file #%.d, named '%s'", file_n, basename(path)))
})
isoreader:::set_finish_file_event_expr({
  isoreader:::log_message(sprintf("finished file #%.d", file_n))
})

c(
  iso_get_reader_example("dual_inlet_example.did"),
  iso_get_reader_example("dual_inlet_example.caf")
) |> iso_read_dual_inlet(read_cache = FALSE)

isoreader:::initialize_options() # reset all isoreader options

Debugging isoreader

The best way to start debugging an isoreader call is to switch the package into debug mode. This is done using the internal iso_turn_debug_on() function. This enables debug messages, turns caching off by default so files are always read anew, and makes the package keep more information in the isofile objects. It continues to catch errors inside file readers (keeping track of them in the problems) unless you set iso_turn_debug_on(catch_errors = FALSE), in which case no errors are caught and stop the processing so you get the full traceback and debugging options of your IDE.

Debugging binary file reads (Isodat)

Errors during the binary file reads usually indicate the approximate position in the file where the error was encountered. The easiest way to get started on figuring out what the file looks like at that position is to use a binary file editor and jump to the position. For a sense of the interpreted structure around that position, one can use iso_print_source_file_structure() which shows what binary patterns isoreader recognized. This binary representation of the source file is only available if the file is read while in debug mode, otherwise file objects would get unnecessarily large:

# turn on debug mode
isoreader:::iso_turn_debug_on()
# read example file
ex <- iso_get_reader_example("dual_inlet_example.did") |>  
  iso_read_dual_inlet(quiet = TRUE)
# retrieve source structure and print a part of it
bin <- ex |> iso_get_source_file_structure() 
bin |> iso_print_source_file_structure(length = 500)

This structure representation shows recognized control elements in <...> and data elements in {...} which are converted to text or numeric representation if the interpretation is unambiguous, or plain hexadecimal characters if the nature of the data cannot be determined with certainty. You can adjust start and length to look at different parts of the binary file or save the the structure to a text file with save_to_file.

For an overview of all the elements (blocks) identified in the binary file as a tibble, use:

bin$blocks |> head(20)

While this provides all elements, the top level structure is provided by the so-called control blocks:

bin$blocks |> dplyr::filter(type == "C block") |> head(20)

To look at specific control-blocks, simply provide the relevant start position to iso_print_source_file_structure():

cdata <- bin$blocks |> dplyr::filter(block == "CData")
cdata

bin |> iso_print_source_file_structure(start = cdata$start, length = 500)

KopfLab/isoreader documentation built on Aug. 6, 2023, 9:22 p.m.