kin.search: Midline and outline tracking over image sequences

Description Usage Arguments Details Value See Also Examples

View source: R/kin.R

Description

Wrapper functions for find.roi that automatically retrieve the contour and midline coordinates of a detected ROI in each image of a sequence through thresholding and segmentation. Also outputs the midline amplitude relative to a reference line determined by an anterior section of the ROI. Supported image formats are jpeg, png, and tiff.

kin.search and kin.simple find the y-value midpoint along the x-value array of the ROI and fits a midline according to a chosen smoothing method (loess or spline). Thus, these assume a horizontal position (see Details).

kin.search and kin.free include arguments for flexible ROI selection.

kin.simple is itself a wrapper for kin.search, finding the largest ROI in field using Otsu thresholding for segmentation.

kin.free does not assume any particular orientation and is intended for finding ROIs freely moving within the image field. This function estimates midlines by various methods and supports parallel processing of frames (see Details).

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
kin.search(
  image.dir = NULL,
  frames = NULL,
  ant.per = 0.1,
  tips = 0.02,
  smooth.n = 0,
  ml.meth = "hull",
  ml.smooth = list("loess", 0.25),
  save = FALSE,
  plot.pml = TRUE,
  out.qual = 1,
  out.dir = NULL,
  ...
)

kin.simple(
  image.dir = NULL,
  frames = NULL,
  ant.per = 0.2,
  tips = 0.02,
  smooth.n = 0,
  ml.meth = "hull",
  ml.smooth = list("loess", 0.25),
  save = FALSE,
  plot.pml = TRUE,
  out.qual = 1,
  out.dir = NULL
)

kin.free(
  image.dir = NULL,
  frames = NULL,
  par = FALSE,
  cores.n = NULL,
  ant.per = 0.1,
  ant.pos = NULL,
  tips = 0.02,
  smooth.n = 1,
  red = NULL,
  ml.meth = "hull",
  ml.smooth = list("spline", 0.25),
  save = FALSE,
  out.qual = 1,
  out.dir = NULL,
  plot.pml = TRUE,
  ...
)

Arguments

image.dir

character, directory containing images to analyze.

frames

numeric, vector indicating which images to process. NULL, the default, will result in all images in image.dir processed.

ant.per

numeric; left-most percentage of ROI that establishes the horizontal reference for the midline displacement.

tips,

numeric, the proportion the the midline data to use in calculation of the head and tail position.

smooth.n,

numeric, the number of contour smoothing iterations. See Details.

ml.meth

character, the midline detection method. One of 'ang' for bisection using free.ml.ang, 'hull' for bisection using free.ml.hull, or 'del' for Delaunay triangulation using free.ml.del. See Details

ml.smooth

a list of length two with unnamed components including a character string specifying the midline smoothing method, either 'loess' or "spline", and a numeric value specifying the amount of smoothing. See Details.

save

logical, value indicating if images should be outputted with midline and predicted midline based on the ant.per lm() overlaying original or binary images.

plot.pml

logical, value indicating if outputted images should include an overlay of the theoretical midline based on ant.per.

out.qual,

numeric, a value between 0-1 representing the quality of outputted images. Ignored if save=FALSE.

out.dir

character, the directory to which outputted images should be saved.

...,

other parameters passed to link{find.roi}.

par

logical, should the frames be processed in parallel using cores.n.

cores.n

numeric, the number of CPU cores to use if par=TRUE. If cores.n=NULL (the default), the total number of cores minus 1 are used.

ant.pos

character, one of NULL, "l","r","u",or "d" to specify the position of the anterior of the ROI. If not NULL, the default algorithm to find the anterior is overridden. See Details.

red

numeric, between 0-1 the proportion of contour coordinates to sample for midline estimates. Ignored if ml.meth is not 'del'. Will speed up midline estimations with Delaunay triangulation. If 'NULL', the full contour retrieved from the ROI will be passed to free.ml.del. See Detail.

Details

The algorithms in kin.simple and kin.search assume a left-right horizontal orientation, i.e., the head of the ROI is positioned left, the tail right. If this is not the case, consider using kin.free or rotating images before processing. The ant.per value therefor establishes the reference line (theoretical straight midline) based on that portion of the head. The midline is calculated as the midpoints between the y extrema for each x position.

kin.search and kin.free choose ROIs based on relative ROI size or position according to find.roi. Parameters for this function are passed through additional arguments with .... Thresholding operations can be performed with an arbitrary (user defined) numeric value or with Otsu's method ('thr="otsu"'). The latter chooses a threshold value by minimizing the combined intra-class variance. See otsu.

kin.simple is more streamlined than kin.search. It attempts to find the largest ROI using Otsu thresholding and invokes other default values of find.roi.

With kin.free, the position of the anterior of the ROI (that which is moving forward in the field) is determined by the displacement of the ROI between the first two frames. Thus, frames must be >1. For analyses of relatively static ROIs in the field (e.g., steadily swimming animals in flumes, etc.), automatically determining the anterior of the ROI may be spurious. In this case, the default automatic determination of the anterior should be overridden by specifying 'l', 'r', 'u', 'd' with ant.pos. These values specify that the anterior region of the ROI is leftmost, rightmost, upmost, or downmost in the field, respectively, and assumes that the origin of the field (0,0) is the upper left corner of each frame.

Midline estimation in kin.free is pursued by one of three algorithms: bisection of contours across the long axis defined by the tips using free.ml.ang or free.ml.hull or by Delaunay triangulation using free.ml.del. The default is 'hull' This choice is not arbitrary. The use of free.ml.ang and free.ml.hull can be faster, but perform poorly for tips that snake back on themselves (i.e., a high degree curvature). The use of free.ml.del can be slower for high resolution outlines, but produces better results when contour regions overlap (i.e, those that snake back on themselves), but produces less precise midlines for complicated contours. Using Delaunay triangulation can be hastened (but possibly with a trade off in precision) by reducing the the complexity of the contour with the 'red' argument. For example, a contour of 1000 coordinates would be reduced to one of 500 with 'red=0.5'.

For midline smoothing, if ml.smooth contains 'spline', smooth_spline from the smoothr package is used to interpolate points between a reduced number of vertices using piecewise cubic polynomials. The number of vertices is calculated based on the number of midline coordinates times numeric value of the list in ml.smooth. If ml.smooth contains 'loess', loess is used to fit a polynomial surface. For contours that have a complicated midline with non-unique x values, say an orginisms swimming vertically in the file, loess smoothing can produce poor results. Thus, spline smoothing is usually the advisable option.

For contour smoothing before midline estimate in kin.free, smooth.n is passed to the smooth.n parameter of free.ml.ang, free.ml.hull, or free.ml.del, which smooths coordinates using a simple moving average. Contours are similarly smoothed in kin.search and kin.simple by invoking coo_smooth from the Momocs package. Users should be wary of oversmoothing by smoothing both the contour (from which the midline is calculated) and the midline.

Value

A list with the following components:

kin.dat a data table consisting of frame-by-frame position parameters for the ROI determined by search.for.

midline A data table containing, for each frame described by frames, the following:

cont A data table containing x and y positions of the contours used to calculate the data in 'kin.dat'. Contains the following:

cont.sm A data table containing the smoothed x and y positions of the contours used to calculate the data in 'kin.dat'. Contains the following:

all.classes A data table containing the following for all ROIs detected:

mid.pred the theoretical midline based on a linear model established by the anterior section of the smoothed midline established by ant.per. Used to calculate midline$wave.y as the orthogonal distance between the line defined by 'x' and 'mid.pred' and each coordinate defined by 'midline$x.sm and midline$y.sm. A data table that contains the following:

dim the x and y dimensions of the images analyzed

See Also

kin.free, find.roi

Examples

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
#### plot caudal amplitude and produce a classic midline waveform plot of a swimming rainbow trout
library(data.table)
##A very long example using kin.search()
## Not run: 

#download example images and place in 'example' subdirectory
f <- "https://github.com/ckenaley/exampledata/blob/master/example.zip?raw=true"

download.file(f, paste0(tempdir(),"/temp.zip"))
unzip(paste0(tempdir(),"/temp.zip"), exdir=tempdir())
unlink(paste0(tempdir(),"/temp.zip"))

dir.create(paste0(tempdir(),"/processed_images"))
kin <- kin.search(image.dir =paste0(tempdir(),"/example"),
      frames=1:50,
      out.dir=paste0(tempdir(),"/processed_images"))
      

#plot instantaneous amplitude of tail (last/rightmost point) over frames
library(ggplot2)
p <- ggplot(dat=kin$kin.dat,aes(x=frame,y=amp))+geom_line()+geom_point()+theme_classic(15)
print(p)

# midline plot
ml <- kin$midline

#leftmost x starts at 0
ml <- ml[,x2:=x-x[1],by=frame]

ml <- merge(ml,kin$kin.dat[,list(frame,amp)],by="frame") #merge these

p <- ggplot(dat=ml,aes(x=x2,y=wave.y))+theme_classic(15)
p <- p+geom_line(aes(group=frame,color=amp),stat="smooth",method = "loess", size = 1.5)
print(p)


## End(Not run)

## A very quick example using kin.simple() and kin.search().

#retrieve image with arguments passed to find.roi()
i <- EBImage::readImage(system.file("extdata/img", "sunfish_BCF.jpg", package = "trackter"))

#create directory and write image to it
t <- tempdir()
dir.create(paste0(t,"/images"))

EBImage::writeImage(i,paste0(t,"/images/sunfish001.jpg"),type = "jpeg")

fi <- list.files(paste0(t,"/images"),full.names=TRUE)
#run kin.search and save output image to directory

kin.srch<- kin.search(image.dir = paste0(t,"/images"),
save = TRUE,out.dir = t,search.for="largest",size.min=0.01)

kin.simp<- kin.simple(image.dir = paste0(t,"/images"),
save = TRUE,out.dir = t)

#plot similar results
library(ggplot2)

kin.both <- rbind(data.table(kin.srch$midline,fun="search"),
data.table(kin.simp$midline,fun="simple"))

qplot(data=kin.both,x=x,y=y.sm,col=fun)

#' #plot midline over original image from kin.simple()
i2 <- EBImage::readImage(paste0(t,"/sunfish001.jpg"))
EBImage::display(i2,method="raster")


##A somewhat long example using kin.free()
#### plot midline waveform on images of swimming ropefish
## Not run: 
library(data.table)
#download example video and establish directories
f <- "https://github.com/ckenaley/exampledata/blob/master/ropefish.avi?raw=true"
download.file(f, paste0(tempdir(),"/ropefish.avi"))

dir.create(paste0(tempdir(),"/images"))
dir.create(paste0(tempdir(),"/out"))
#extract images
vid.to.images(paste0(tempdir(),"/ropefish.avi"), out.dir = paste0(tempdir(),"/images"))

#run kin.free() 

kin <- kin.free(image.dir =paste0(tempdir(),"/images"),
      par=TRUE,
      save=TRUE,
      out.dir=paste0(tempdir(),"/out"),
      ml.smooth=list("spline",0.5),
      thr = "otsu",
      ml.meth="ang",
      ant.pos="l",
      red=0.5,
      size.min=0.01
      )


#see results
#images
fi <- list.files(paste0(tempdir(),"/out"),full.names=TRUE)
EBImage::display(EBImage::readImage(fi[320]),"raster")

#with gg.overlay() on first 300 frames

gg.overlay(kin=kin,
under="cont.sm",
over="midline",
frames=0:299,
size=.2,
animate=TRUE,
zoom=FALSE,
alpha=0.01,
col="red",
fps=10)




## End(Not run)

ckenaley/trackter documentation built on Feb. 11, 2022, 6:43 a.m.