events_v2: Pipeline function to extract events

In mpio-be/SNB2: Smart Nest-box Data Management

Pipeline function to extract events

Description

Calls a set of sub-functions to extract information about box entries and exits. Note that all activity that is not connected with any transponder and triggers only one light barrier is removed. The pros and cons of different parameter choices can be assessed via SNBvsVid_v2, which performs a comparison with existing video data during provisioning. Note: DATA IS MADE UNIQUE FOR NOW (excepting r_pk or course)! Please make sure that you only supply one box at a time!

Usage

events_v2(
  x,
  setTZ = "Etc/GMT-2",
  group_ins_and_outs = TRUE,
  FUN = "translate_validity_v2",
  tr_threshold = 5,
  broken_LB_threshold = 600,
  cluster_events_threshold = 2,
  max_distance = 16 * 60 * 60,
  cluster_fronts_threshold = 5,
  no_front = NULL,
  silent = FALSE
)

Arguments

x	A data.table with raw data as provided by the function fetch_data_v2
group_ins_and_outs	logical. Should every piece of activity be returned separately (e.g. to merge with observational data), or should they be grouped together to "INs" and "OUTs"? Defaults to TRUE.
FUN	function to translate detailed information about the direction of the activity. Defaults to the function translate_validity_v2, which assigns "IN-OUT" and "FRONT"
tr_threshold	Optional argument passed to function fetch_ins_outs_v2. Time in seconds where the light barriers are assigned to a specific transponder. Defaults to 5.
broken_LB_threshold	Optional argument passed to function fetch_ins_outs_v2. Time in seconds after which a light barrier that is constantly on will be marked as "OFF". Defaults to 600. Experimental.
cluster_events_threshold	Optional argument passed to function fetch_ins_outs_v2. Time in seconds within which different pieces of activity will be considered as belonging to a single event. Defaults to 1. Note this parameter may reduce the computing time at the cost of detail.
max_distance	Optional argument passed to function concat_events_v2. Maximum time in seconds within which two pieces of activity may be grouped together. Defaults to 14 hours.
cluster_fronts_threshold	Optional argument passed to function combine_fronts_v2. Time in seconds within which "FRONT" events of the same transponder will be grouped together.
no_front	NULL or character(). Optional argument passed to function combine_fronts_v2. Defines which directions are assumed to be a "FRONT" when events are grouped by the function combine_front_v2. Defaults to NULL, which means that everything where no pass through the nestbox opening was registered is scored as a "FRONT".
silent	Should the stats about the data quality be printed or not? Defaults to TRUE.

Details

Each subfunction performs a specific task. The tasks can be described as follows:

1. fetch_ins_outs_v2: Separate different pieces of activity and assign the corresponding transponder. Consecutive rows are assigned to different pieces of activity, if both light barriers are off for a certain time. This time cutoff is defined by the variable 'cluster_events_threshold'. Note that the data is split only by the light barrier information, if cluster_events_threshold=0. Transponders are then assigned to any event within "tr_threshold" seconds of a transponder reading. If a light barrier is constantly ON (e.g. because of dirt), it is set to OFF after broken_LB_threshold seconds (experimental!).

2. assign_direction_v2: Remove any event that has only activity at one light barrier and no transponder connected to it (for example FRONT or BACK without transponder). Then, fetch time difference between conscutive pieces of activity and assign a transponder to them, if possible. Assign which light barrier is triggered first (=position of the bird at the beginning of the activity: o means "outside", i means "inside"; "O/I" therefore means that the event started on the outside and ended on the inside and therefore is an "IN") and which light barrier is triggered last (=position of the bird at the end of the activity) (Note that the light barriers have a pulse every 50 milliseconds; if both lightbarriers are triggered within 50 ms, their order cannot be distinguished, and the hardware_threshold should therefore be kept at 50 ms= 0.05s. In this case the position of the bird is unclear = "?") If more than one transponder is connected to one event, duplicate the event and set the beginning and end directions to "unclear".

3. concat_events_v2: Combine activities that belong together based on the transponder reading, and on the light barrier information using the following rulse:

   • If activity starts with the bird being outside, never combine with the previous event.

   • If activity ends with the bird being outside, never combine with the previous event.

   • If the start position of the bird is unclear, only combine if the previous activity ended with an "inside". Mark the start position with a lower-case.

   • If the end position of the bird is unclear, only combine if the next activity starts with an "inside". Mark the start position with a lower-case. Remove redundant information.

4. combine_front_v2: Assign transponder to FRONTs without transponder if they are within cluster_fronts_threshold seconds of a transponder reading. This is implemented because FRONT events preceding an IN of a transpondered bird are often very short and often the transponder is not noted although when looking at the data the identity of the bird is easy to see.

5. translate_validity_v2: Function that assigns easy-to-interpret directions (e.g. "IN-OUT") to the detailed but confusing directions given by the light barriers (e.g. "o/I|I/O"). The default function uses the following rules (in the specified order):

   1. assign IN-OUT to all lines (note that some of this will be over-written by the next rules!)

   2. make everything where only a single activity is recorded (e.g. concat_events_v2 did not combine 2 or more events!) as FRONT unless...

   3. unless it is I/O (=OUT) or O/I (=IN) (based on the video data) or ...

   4. has Os only (=IN) <- based on video material, 90

Value

• If group_ins_and_outs = TRUE: data.table of SNB events with 10 columns. Use plot(this_datatable) to immediately plot this data. Use attr(this_datatable, "stats") to fetch information on the data quality (use print() for readability).

  • box: box number

  • transp: transponder connected to the event

  • in_: datetime of the start of the activity

  • out_: datetime of the end of the activity

  • in_duration: how long did the activity in column "in_" take (e.g. the bird might have spent 5 seconds in the nest opening before entering properly)

  • out_duration: how long did the activity in the out_ column take?

  • in_r_pk: link to the raw bxxx tables: r_pk of in_

  • out_r_pk: link to the raw bxxx tables: r_pk of out_

  • direction_detail: Full information about the direction of movements as specified by the light barriers. Formatted e.g. as O/I|I/O, containing the information (position of bird at the beginning of the "in_")/(position of the bird at the end of the "in_")|(position of bird at the beginning of the "out_")/(position of the bird at the end of the "out_"). Capital letters mean that this is true light barrier information, lower-case letters mean that the position of the bird was inferred by the next or previous activity (e.g. and OUT is usually followed by an IN).

  • direction: Summarized directional information as specified by the function translate_validity_v2, e.g. "IN" or "OUT". Note that this function can be supplied as an argument to the function, so that it is easy to adjust to any needs.

If group_ins_and_outs = FALSE:

A data.table with the following information:

• box: The box number

• r_pk: The r_pk referring to the original bxxx table.

• path: The path the the raw data file

• startt: The start of the activity (rounded to the nearest second)

• endt: The end of the activity (rounded to the nearest second)

• transp: The transponder

• direction_detail: Detailed directional information, where the first letter indicates where the event started (O = outside, I = inside, ? = unclear), and the second letter where is ended. "I/O" would for instance mean "movement from the inside to the outside" -> a bord left the nest box.

• direction: Easy-to-interpret directional information. Note that this assignment is a rough estimate.

Note

Please supply only one box at a time! There is a wrapper function which can be used to fetch data across the two databases and across boxes: eva

Author(s)

LS

See Also

eva

Examples

##Not run
#establish connection
con = dbcon() #or other connection to database

#fetch raw data for a single box
#note that this function should ALWAYS only be run on data from a single box!
raw_x = dbq(con, 'SELECT * FROM SNBatWESTERHOLZ_v2.b001 
  WHERE datetime_ > "2019-05-01" and datetime_ < "2019-05-31"')
raw_x[, box := 1] #there is no need to supply the box number, but you may need it lateron.

#Calculate when individuals were at the box (e.g. "IN-OUT")
x = events_v2(raw_x)
plot(x)
print(attr(x, 'stats'))  

#fetch data where INs and OUTs are not yet combined 
x = events_v2(raw_x, group_ins_and_outs = FALSE)
x #note that there is no plot method for this output.

#fetch data for multiple boxes
X = list()
for(i in c(3, 5, 206, 210)){
 print(i)
 raw_x = dbq(con, paste0('SELECT * FROM SNBatWESTERHOLZ_v2.', int2b(i), 
               ' where datetime_ > "2019-05-01" and datetime_ < "2019-05-31"'))
 x = events_v2(raw_x, silent = TRUE)
 x[, box := i]
 X[[length(X)+1]] <- x
}


#Plot method for SNB data (note that data has class "SNBoutput")
plot(X[[1]])
plot(X[[3]])

#combine to one dataset
all = rbindlist(X)

#use all cores -- 5 minutes, depending on the settings
#use different parameter settings to adjust daa to your needs
a = Sys.time()
library(doParallel)
cl = makePSOCKcluster(2) # 50 cores
registerDoParallel(cl)

d = foreach(i = 1:277) %dopar% {
 require(SNB2)
con = dbcon() #or other connection to database
 raw_x = dbq(con, paste0('SELECT * FROM SNBatWESTERHOLZ_v2.', int2b(i), 
           ' where datetime_ > "2019-05-01" and datetime_ < "2019-05-31"'))
 closeCon(con)
 x = events_v2(raw_x, silent = TRUE, tr_threshold = 2, cluster_events_threshold = 2, 
           max_distance = 16*60*60, cluster_fronts_threshold = 5)
 x[, box := i]
}
stopCluster(cl)
registerDoSEQ()
closeAllConnections()
Sys.time()-a

stats = as.data.table(do.call(rbind,lapply(d, FUN = function(x) attr(x, "stats"))))
setnames(stats, names(stats), c("dupl", "max_tr", "mean_tr", 
         "prop_reliable", "prop_reliable_tr"))

hist(stats[, prop_reliable])

mpio-be/SNB2 documentation built on Sept. 17, 2024, 4:27 p.m.