Retrieving a single frame from a video or camera stream is fast, but it is not
instantaneous because Rvision
(and therefore OpenCV
in the background) needs
to grab and decode each frame before it can be used for further processing. For
most applications, that small time penalty is not an issue; losing a few
milliseconds per frame will not be felt by a user processing short and/or low
resolution videos for instance.
However, the frame decoding time will increase significantly with the resolution
of the video/camera stream; and for applications requiring the processing of
long videos or live camera feeds, the total time lost retrieving frames will
quickly increase. Moreover, the process of retrieving frames is blocking: while
Rvision
grabs and decodes a frame, it cannot work on other frames that were
previously captured.
A solution to this issue is to make use of the multi-tasking ability of most modern computers to perform the frame retrieving process in parallel with the rest of the image processing. The principle is fairly simple: one processing thread (thread 1) is in charge of grabbing, decoding, and storing frames in a shared dynamic queue (or buffer); a second thread (thread 2) is in charge of processing these pre-loaded frames further; while thread 2 is working on a frame, thread 1 can keep filling up the queue with new frames so that thread 2 never has to stop and wait for a new frame to be retrieved. On a multi-core/processor computer (most computers nowadays), thread 1 and 2 can be operated in parallel on separate cores/processors, effectively reducing the frame waiting time to near-zero for thread 2 (but see caveats section below).
Unfortunately, R
is a single-threaded language, meaning that it cannot
natively create and operate multiple parallel processing threads. Fortunately,
the C++
language that is used in the background
by Rvision
can do it. We used that ability - and some precautions to avoid
memory access conflicts with R
- to create the Queue
class of objects which
job is to pre-load in memory video and camera stream frames without blocking the
execution of the main (and unique) R
processing thread, and then give the
main R
thread near-instantaneous access to these pre-loaded frames when
required. Once a frame has been collected by the main R
thread, it is removed
from the queue to make space for another frame.
Queue
objects are created using the queue()
function. For instance:
# Find the path to the Balloon.mp4 video provided with Rvision path_to_video <- system.file("sample_vid", "Balloon.mp4", package = "Rvision") # Open the video file stream my_video <- video(filename = path_to_video) # Create a queue of frames my_buf <- queue(my_video, size = 10, delay = 1000, overflow = "pause")
The queue()
function can take 4 parameters:
x
corresponds to the source of the frames to be queued. It should either be
a Video
or a Stream
object. size
corresponds to the number of frames that the queue can store at any one
time. By default, a Queue
object will be able to hold 10 frames at once. If
you increase the value of the size
parameter, the Queue
object will use up
more RAM as a result, so be mindful of your computer's resources. delay
corresponds to the time (in microseconds) between two queue update
cycles. During an update cycle, the Queue
object checks whether it is full or
not and whether frames have been collected by the main R
thread. If it is not
full, it retrieves a new frame from the source and stores it; if frames have
been collected by the main R
thread, it removes them from the queue; it then
waits the duration set by delay
before starting a new update cycle. Reducing
the value of delay
will increase the frequency of the update cycles but will
also increase the computational load of the core/processor running the queuing
thread.overflow
corresponds to the behavior the Queue
object should adopt once it
is full. By default, the queuing process will "pause" (that is stop retrieving
new frames from the source and storing them) until a frame is collected by the
main R
thread. This is the behavior of choice for video processing as it does
not skip any frame while ensuring that the queue storage memory does not grow
more than what the user wants. The queuing process can also "replace" the oldest
frame in the queue with a new one. This is usually a good choice for processing
live camera stream as it ensures that the queue storage memory does not grow
more than what the user wants. However, frames may be skipped if the main R
thread cannot collect and process the frames faster than they can be retrieved
and stored by the queuing thread. Finally, the queuing process can "grow" the
queue by doubling its size each time it fills up. This behavior allows for not
pausing the retrieval process and avoids frame skipping but it is very much NOT
recommended unless you know what you are doing. Indeed, this can lead to
excessive RAM usage and decreased computing performance across the board. Once a Queue
object is created, it starts immediately filling up with images
retrieved from the Video
or a Stream
source object. Note that, if you had
previously read frames from a Video
source object before creating the Queue
object, the latter will start retrieving frames from where you left off (e.g. if
you have already read the first 10 frames, the queue will start filling up from
the 11th).
Once a Queue
object is not required anymore, it can be released from memory as
follows:
release(my_buf)
Queue
objectThe main purpose of a Queue
object is to pre-load and store frames for fast
access later on. The pre-loading and storing happens by itself in the background
so you do not need to take care of this. Collecting a frame from the queue into
the main R
thread can be done using the readNext()
function. For example:
# Collect the next available frame from the queue and store it in a new # Image object frame <- readNext(my_buf) # Collect the next available frame from the queue and store it in an existing # Image object readNext(my_buf, target = frame)
At any time, you can check the state of the queue as follows:
# Is the queue empty? empty(my_buf) # Is the queue full? full(my_buf) # What is the current number of frames in the queue? length(my_buf) # What is the maximum number of frames that the queue can hold? capacity(my_buf) # What is the index of the next frame available? (for video queues only) frame(my_buf) # What are the dimensions of the queue? dim(my_buf) nrow(my_buf) ncol(my_buf)
Queue
objectsQueue
objects can be very useful to speed up the processing of camera streams
and long videos, especially if their resolution is high (HD or above). For short
videos or low resolution videos and camera streams, the benefits of using a
Queue
object will be very limited because retrieving frames from such sources
is already very fast.
If the processing of the frames on the main R
thread is much faster than the
retrieving and storing of the frames inside a Queue
object, you risk emptying
the queue faster than in can fill up. If that is the case, readNext()
will
display a warning to this effect but will not return an error (as it would, for
instance, when reaching the end of a video file). In addition, it will not
create a new image or modify the target image. You will need to make sure that
your code can catch these warnings and act accordingly (e.g., wait a turn before
trying again). You can reduce the risks of the queue becoming empty by (1)
increasing the size of the queue when creating it, (2) making sure the queue has
completely filled up before starting the processing of the frames, and (3)
reducing the delay between to queue updates. In any case, Queue
objects are
better suited for and more beneficial in cases where the processing of the
frames is as slow or slower than the frame retrieving process.
Finally, Queue
objects read frames from existing Video
and Stream
objects.
This means that they modify the state of these source objects the same way a
user would by reading directly from them. This has a few of consequences that
require some attention when using Queue
object. First, a Queue
object will
start reading the source object from whichever state you have left it off. For
instance, if you have read the first 10 frames of a video before passing it to a
Queue
object, the Queue
object will start reading it from the 11th frame.
Conversely, if you pass a newly created Video
object to a Queue
object with
a size of 10, the Queue
object will immediately start reading and storing the
first 10 frames. If you then decide to read a frame directly from the Video
object, it will return its 11th frame, not its first one. Therefore, to avoid
competing with a Queue
object on a given Video
object, it is strongly
recommended to avoid creating code that mix reading frames from a Queue
object
and reading frames from the Video
object that Queue
object is using.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.