Creating a `BGF` from a Custom Fermentation System

In bgfanalyzer: Analyze Microbial Biogas Fermentation Data

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

Whats this vignette is about

This vignette is the second of three, that focus the topic of how to get data into a BGF. Assuming the reader has read vignette("Introducing-BGF",package="bgfanalyzer") and vignette("BGF-commercial",package="bgfanalyzer") and is familiar with the general concepts of BGF's and how to build a BGF from a report generated by a distinct commercial fermentation system (see vignette("BGF-commercial",package="bgfanalyzer") for details).

In this vignette, we will discuss how to build a BGF from one or more external files, that are in a standardized text format. Such files are expected to contain at least cumulative exhaust gas volume measurements and a time log of a single fermentation, but can include additional sensor readings such as pH or gas quality measurements.

In the following sections we will see how to build a BGF from multiple such standardized files, which store the data of several fermentations.

Background

The data used during this vignette was acquired from a custom fermentation system. It consisted of a 2l stirred and heated batch reactor. This reactor was equipped with several sensors, such as a pH-, temperature- and RedOx-electrodes, allowing continuously monitoring of essential fermentation parameters. In addition, the cumulative exhaust gas volume was recorded and the exhaust gas composition (= gas quality) was sequentially analyzed.

The data files used in the following, can be grouped in two groups:

The Fermentation_*-files store cumulative exhaust gas volumes measurements together with other sensor readings of each one fermentation. The gasq_*-files store gas quality measurements of the respective fermentations.

The fermentations were carried out in individually heated batch reactors and sensor readings within Fermentation_*-files were recorded by third party software. The data in gasq_*-files originates from a gas chromatograph sequentially analyzing the gas composition within the exhaust gas of the same fermentations.

Sensor readings in Fermentation_*-files were manually converted into the standard format using an R script specifically developed to convert text reports generated by the third party software. Gas quality measurements were collected using instrument specific software and uploaded into a database afterwards. The gasq_*-files were extracted from that database and were already in the standardized format upon database extraction.

Object creation

Now, let's start building a simple BGF form an external file in standard format. This BGF will not contain any 'Blank' fermentations.

We will use the function from_standard_record() to create a new BGF.

# load library
library(bgfanalyzer)

# We build a BGF from
myBGF<-from_standard_record(ReactorLayout = "Substrat A",
                     ProcessTemp = 80,
                     InocToSubRatio = 0.1,
                     path = system.file("extdata","Fermentation_A.tsv",package = "bgfanalyzer"),
                     time_col = 1,
                     product_col = 3,
                     BlankLabel = "Blank",
                     name = "myBGF",
                     units = "days")
# inspect object
myBGF

# have a closer look at its 'metaData'-layer
myBGF$metaData

A new BGF was created with only one row in the metaData-layer!

How about the BioGasData-layer? Let's take a look at its head() and tail() output:

# look at the first six rows of the 'BioGasData'-layer
head(myBGF$BioGasData)

# look at a snippet of the last six rows of the 'BioGasData'-layer
tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(2,1,0)))])

As we can see, reactor, time and product columns were already filled with data from the external file and in addition eight new columns with data were added to the BioGasData-layer of 'myBGF' through the call to from_standard_record()

Adding additional data to BioGasData-layer

Now we will add the gas composition measurements stored in gasq_A.tsv. First, we import the file, next we add it to the BGF.

# import the gas quality measurements stored in a separate file
gasq_A <- import_standard_record(ipath = system.file("extdata","gasq_A.tsv",package = "bgfanalyzer"),
                                 dec = ".",
                                 sep = "\t",
                                 header = TRUE,
                                 mkFRTime = "2025-05-11 14:08:35",
                                 FRTime_col = 1,
                                 units = "days")

# add gas quality measurements to BGF
myBGF <- add_BG_parameter(myBGF,
                          parameter = gasq_A,
                          reactor = "R1",
                          time = 3,
                          value = 2,
                          name = "H2",
                          cut_zero = TRUE,
                          interpolate_missing = FALSE)

# look at a snippet of the first six rows of the 'BioGsaData'-layer to see the results
head(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))])

A new column, H2, storing the amount of the target gas in the exhaust gas (in this example the target gas is Hydrogen, H~2~) was added to the BioGasData-layer.

However, most of the entries in this new column are NA and insertion of data into the existing BioGasData-layer created additional NA's whenever a value was inserted:

# data insertion created gaps
myBGF$BioGasData[c(97:99),c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))]

NA removal through data inter- and extrapolation

Such 'gaps' in the data can be closed using the function na_correction(). By default, this function will close the gaps in all numeric columns of the BioGasData-layer, as long as there is a value before and a value after the gap.

# close the gaps
myBGF <- na_correction(myBGF)

# look at a snippet of the first six rows
head(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))])

# look at the gap that arrose from data insertion
myBGF$BioGasData[c(97:99),c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))]

# look at a snippet of the last six rows
tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))])

The data is almost free of NA's now. Only in the end of the BioGasData-layer, NA's in H2 still exist. We can handle this by a second call to na_correction() with some adjustments.

We will specifically correct column H2 and will not end correction if no value after the gap exist. To avoid that this NA-correction yields in negative concentrations, we add argument sub_zero = 0 to the function call:

# correct remaining NA's in H2
myBGF <- na_correction(myBGF,
                       which = "H2",
                       end=F,
                       sub_zero=0)

# insepct results
tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))])

As we can see, the terminal NA's in H2 have been replaced by interpolated values.

Further data processing

Now, we have build a BGF from data stored within two external files in a standardized data format. This BGF still has missing values in production, rel_production, net_product and yield of its BioGasData-layer, which we need to take care of.

First, we calculate the values for production using calculate_flow_from_volume():

# cumulative exhaust gas volume mesurements in 'product' can be used to calculate 
# the 'production', a.k.a the biogas flow
myBGF <- calculate_flow_from_volume(myBGF)

# inspect standard columns of 'BioGasData'-layer
head(myBGF$BioGasData[c(1:7)])

Similarly, we can calculate values for rel_production using relative_production():

# calculate relative production
myBGF <- relative_production(myBGF)

# inspect standard columns of 'BioGasData'-layer
head(myBGF$BioGasData[c(1:7)])

Let's proceed with the calculation of net_product. Therefore, we use the function netGasGC(), which takes the values of production and multiplies it with the target gas concentration stored in H2, to calculate the net amount of target gas produced in between observations. Next, this function will cumulate these 'net production' values which yields in the final net_product values.

# calculate net_product
myBGF <- netGasGC(myBGF,
                  purity = "H2",
                  substract_blank = FALSE)

# inspect standard columns of 'BioGasData'-layer
tail(myBGF$BioGasData[c(1:7)])

We successfully calculated net_product. Before we continue calculating the yield, we must first add some information to the metaData-layer. The yield reflects the amount of net_product produced per 'input organics'. This example fermentation was conducted in a 3L reactor with 3.23 wt.\% organic total solutes (oTS), which makes 96.89 g oTS in the fermentation. We add this value like this:

# add the oTS to 'metaData'-layer
myBGF <- add_metaData(myBGF,98.89,lab = "oTS")

# inspect change in 'metaData'-layer
myBGF$metaData

Now, we can calculate the yield like this:

# calculate 'yield'
myBGF <- calc_yield(myBGF,pos = 4)

# inspect standard columns of 'BioGasData'-layer
tail(myBGF$BioGasData[c(1:7)])

# get the yield summary
myBGF <- summarize_yield(myBGF)

# inspect change in 'metaData'-layer
myBGF$metaData

Afterwards, all standard columns of the BGF were calculated and we can use bgf_plot() to visualize the data:

# build all standard plots
bgf_plot(myBGF)

In the line plots, we can see some disturbance in the beginning of the fermentation. We can 'correct' this by moving the '0' value in the observation time to the right, and afterwards delete all data with a 'negative' observation time. A suitable new '0' value can be found interactively:

# print 'product_curve' interactively
bgf_plot(myBGF,type = "product",interaction=TRUE)

Now we can hover with the courser over the line and identify 'time=0.104' as the first value after the disturbance. Alternatively, we could have use `View(myBGF\$BioGasData)` (or `print(myBGF\$BioGasData)`) and search for a suitable time there.

So, data clean up is possible by deleting the first 0.11 days of this 6 days fermentation. By doing this, 98 \% of the data will be kept.

# trim the fermentation
myBGF<-trim_FR_time(myBGF,0.11)

# inspect results by ploting
bgf_plot(myBGF,type = "product")

When shifting the '0' value to the right, it is advised to run update_BGF() to ensure the internal logic of the BGF and to rerun data processing functions (calculate_flow_from_volume(), relative_production(), netGasGC(), calc_yield() and summarize_yield()).

So:

# ensure internal logic of the 'BGF'
myBGF<-update_BGF(myBGF)

# re-calculate 'production'
myBGF <- calculate_flow_from_volume(myBGF)

# re-calculate 'rel_production'
myBGF <- relative_production(myBGF)

# re-calculate 'net_product'
myBGF <- netGasGC(myBGF,"H2",substract_blank = FALSE)

# re-calculate 'yield'
myBGF <- calc_yield(myBGF,pos = 4)

# plot 'BGF' again
bgf_plot(myBGF)

Let's try printing the BGF:

# print the BGF
myBGF

Finally, we have a complete BGF with data of a single fermentation. The input data of that BGF was split over two external data files. In the following, we will see, how to add further fermentations with the same, or a different reactor layout to that BGF.

Adding further fermentations

Now, that a final BGF is build let's see how to add another fermentation with the same reactor layout.

Same reactor layout

Like the first fermentation, also the input data of the second is split in two external files. The first step is importing these files and adding the new data to the existing BGF:

# Import the new data directly from the input file with cumulative exhaust gas volume measurements
myBGF <- add_standard_record(myBGF,
                             path = system.file("extdata","Fermentation_B.tsv",package = "bgfanalyzer"),
                             RName = "R2",
                             time_col = "UTC",
                             units = "days",
                             product_col = "GCounter..ml.")

# updating  internal logic is highly recommended
myBGF <- update_BGF(myBGF)

# correct 'metaData$Layout' for new fermentation
myBGF <- alter_whatever(myBGF,layer = "metaData",what = "Layout",value = "Substrate A")

# already add 'metaData$oTS' at this point
myBGF <- alter_whatever(myBGF,layer = "metaData",what = "oTS",value = 98.89,ID = "R2")

# import respective gas quality measurements
gasq_B <- import_standard_record(ipath = system.file("extdata","gasq_B.tsv",package = "bgfanalyzer"),
                                 dec = ".",
                                 sep = "\t",
                                 header = TRUE,
                                 mkFRTime = "2025-05-19 22:00:00",
                                 FRTime_col = 1,
                                 units = "days")

# add gas quality measurements to BGF
myBGF <- add_BG_parameter(myBGF,
                          parameter = gasq_B,
                          reactor = "R2",
                          time = 3,
                          value = 2,
                          name = "H2",
                          makeCol = FALSE,
                          cut_zero = TRUE,
                          interpolate_missing = TRUE)

# look at a snippet of the first six rows of the 'BioGsaData'-layer to see the results
tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))])

Now that we have added the raw data of a second fermentation we can interactively plot the product curve to see if (and how) we need to trim the added data:

# plot interactive product curve of BGF
plot_product_curve(myBGF,interaction=TRUE)

The data of 'R2' look strange from 0 to 0.61 d and after 2.49 d. We can adjust this like before using

# remove data later than 2.49d
myBGF <- trim_FR_time(myBGF,
                      value = 2.49,
                      mode = "R2",
                      left_end = FALSE)

# remove data before 0.61d
myBGF <- trim_FR_time(myBGF,
                      value = 0.61,
                      mode = "R2")

# plot product curve again to inspect results
plot_product_curve(myBGF)

The data looks much better now! Let's continue with calculating production, rel_production, netGas and yield for the newly added fermentation.

We can use the same functions as before:

# ensure internal logic of the 'BGF'
myBGF<-update_BGF(myBGF)

# close gaps
myBGF <- na_correction(myBGF)

# re-calculate 'production'
myBGF <- calculate_flow_from_volume(myBGF)

# re-calculate 'rel_production'
myBGF <- relative_production(myBGF)

# re-calculate 'net_product'
myBGF <- netGasGC(myBGF,"H2",substract_blank = FALSE)

# re-calculate 'yield'
myBGF <- calc_yield(myBGF,pos = 4)

# summarize yield
myBGF <- summarize_yield(myBGF)

# print the new BGF
myBGF

In the print() we see, the BGF now has 2 fermentations. In the 'yield summary' standard deviations for 'yield', 'production' and 'time_production' are calculated as the fermentations both have 'Layout' 'Substrate A'.

Interestingly, although the observation times of these example fermentations differ, the 'yield' at the final observation time is quite similar as seen in the low 'sd_yield' value.

Different reactor layout

Now that we have build a BGF with two fermentations of the same reactor layout from several external files, it is time to add further fermentations, that have a different reactor layout.

At first we create a second BGF with two fermentations of reactor layout 'Substrate B', and next we merge the two BGF's into a single object.

Let's start with creating the new BGF. It will have 2 fermentations in the metaData-layer, and the raw data of the first fermentation will be directly imported to the BioGasLayer-layer.

# create a new BGF directly from the record of a third fermentation
myBGF2<-from_standard_record(ReactorLayout = c("2*Substrate B"),
                             ProcessTemp = 80,
                             InocToSubRatio = 0.1,
                             path = system.file("extdata","Fermentation_C.tsv",package = "bgfanalyzer"),
                             time_col = 1,
                             product_col = 3,
                             BlankLabel = "Blank",
                             name = "myBGF2",
                             units = "days")

# import respective gas quality data
gasq_C <- import_standard_record(ipath = system.file("extdata","gasq_C.tsv",package = "bgfanalyzer"),
                                 dec = ".",
                                 sep = "\t",
                                 header = TRUE,
                                 mkFRTime = "2024-10-27 05:30:00",
                                 FRTime_col = 1,
                                 units = "days")

# add gas quality measurements to BGF
myBGF2 <- add_BG_parameter(myBGF2,
                          parameter = gasq_C,
                          reactor = "R1",
                          time = 3,
                          value = 2,
                          name = "H2",
                          cut_zero = TRUE,
                          interpolate_missing = TRUE)


# print the new BGF
myBGF2

Before, further processing the data, let's add the raw data from the last fermentation:

myBGF2 <- add_standard_record(myBGF2,
                             path = system.file("extdata","Fermentation_D.tsv",package = "bgfanalyzer"),
                             RName = "R2",
                             time_col = "UTC",
                             units = "days",
                             product_col = "GCounter..ml.")

# updating  internal logic is highly recommended
myBGF2 <- update_BGF(myBGF2)

# import respective gas quality data
gasq_D <- import_standard_record(ipath = system.file("extdata","gasq_D.tsv",package = "bgfanalyzer"),
                                 dec = ".",
                                 sep = "\t",
                                 header = TRUE,
                                 mkFRTime = "2024-10-30 23:00:00",
                                 FRTime_col = 1,
                                 units = "days")

# add gas quality measurements to BGF
myBGF2 <- add_BG_parameter(myBGF2,
                          parameter = gasq_D,
                          reactor = "R2",
                          time = 3,
                          value = 2,
                          name = "H2",
                          makeCol = FALSE,
                          cut_zero = TRUE,
                          interpolate_missing = TRUE)

# print the BGF
myBGF2

As we can see, the number of observations in the BioGasData-layer has increased indicating a successful data addition.

We continue with processing the data. First, we check if the data needs trimming and afterwards we can head towards yield calculation!

# interactively plot product curve
plot_product_curve(myBGF2,interaction=TRUE)

We see that both fermentations need adjustments in the beginning. 'R1' will be trimmed by 0.42d and 'R2' will be trimmed by 0.56d counting from observation start. Furthermore, 'R1' needs to be cut after 4.08d, which will be our first step:

# trim 'R1' fermentation
myBGF2 <- trim_FR_time(myBGF2,4.08,"R1",left_end = FALSE)

# updating  internal logic is highly recommended
myBGF2 <- update_BGF(myBGF2)

# trim 'R1' fermentation
myBGF2 <- trim_FR_time(myBGF2,0.42,"R1")

# updating  internal logic is highly recommended
myBGF2 <- update_BGF(myBGF2)

# trim 'R2' fermentation
myBGF2 <- trim_FR_time(myBGF2,0.56,"R2")

# updating  internal logic is highly recommended
myBGF2 <- update_BGF(myBGF2)

# plot the product curve again to see results
plot_product_curve(myBGF2)

Now we can start the data processing loop:

# updating  internal logic is highly recommended
myBGF2 <- update_BGF(myBGF2)

# close gaps in data
myBGF2 <- na_correction(myBGF2,end=F)

# calculate production
myBGF2 <- calculate_flow_from_volume(myBGF2)

# calculate relative production
myBGF2 <- relative_production(myBGF2)

# calculate net gas
myBGF2 <- netGasGC(myBGF2,"H2",substract_blank = FALSE)

# add a oTS column at the metaData-layer
myBGF2<-add_metaData(myBGF2,c(85.5,85.5),lab="oTS")

# calculate yield
myBGF2<-calc_yield(myBGF2,4)

# summarize yield
myBGF2<-summarize_yield(myBGF2)

# print BGF
myBGF2

Finally, we can merge the two BGF's into a new BGF containing all 4 fermentations using merge_BGF() :

# merge the two BGFs
mergedBGF<-merge_BGF(myBGF,myBGF2,name = "merged BGF")

# print the merged BGF
mergedBGF

Now we have successfully created a single BGF with 4 fermentations of two reactor layouts! The raw data for this BGF was extracted from 8 external files.

Note that it would have been possible to create the same object without merging two BGF's. To do this one could set e.g. 'ReactorLayout = c("2*Substrate A","2*Substrate B")' in the initial call to from_standard_record() to set up a BGF for 4 fermentations and than successively add further data from external files like described e. g. for myBGF2 in the section above.

bgfanalyzer documentation built on Sept. 26, 2026, 5:07 p.m.