In Sage-Bionetworks/mhealthtools: mhealthtools - Package for processing mobile sensor data and to extract meaningful features

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

mhealthtools was built to help people study the physiological symptoms that can be tracked through the sensors of the modern day smartphone. This can be the tremor in your hand using the accelerometer and gyroscope sensors while holding the phone in your hand; or Bradykinesia and Dyskinesia through the acceleremeter and gyroscope sensors while having the phone in the front pocket of your pants as you walk; or heartrate using the camera with your finger against it. Whatever the experiment design (activity), if it uses smartphone sensors, you can use mhealthtools to extract useful and sometimes physiologically meaningful features.

Here we describe the use of mhealthtools for certain modules, that are taken from the mPower study run by Sage Bionetworks. For further reading please refer to the article published in nature.

Let's load the package before we proceed

library(mhealthtools)

library(dplyr)

Walk Module

In this activity, the participant is asked to keep the phone in his/her pants' front pocket (if not in the waistband of the pants) and then walk in a straight line for some fixed amount of time (e.g. 30s). For someone with Parkinson's we expect to see symptoms of Dyskinesia and Bradykinesia.

The features extracted from get_walk_features do capture to some extent the effects of those symptoms. In the default processing, the signal is broken down into two IMFs (Intrinsic Mode Functions) using the Hilbert Huang Transform before being analyzed.

mhealthtools comes with preloaded sample walk data from the mpower study. Let's take a look at the walk data before we proceed to analyze it.

dplyr::glimpse(walk_data)

Let's look at the accelerometer data -- userAcceleration (plotting only the first ten seconds)

library(ggplot2)
a <- walk_data$userAcceleration
a$t <- walk_data$timestamp
a <- tidyr::gather(a, 'axis' , 'value', -t)
a <- a %>% dplyr::filter(t<10)
ggplot(a, aes(x = t, y= value)) +
          geom_line() +
          facet_wrap(~axis, dir = 'v', strip.position = 'right')

Let's look at the gyroscope data -- rotationRate (plotting only the first ten seconds)

library(ggplot2)
a <- walk_data$rotationRate
a$t <- walk_data$timestamp
a <- tidyr::gather(a, 'axis' , 'value', -t)
a <- a %>% dplyr::filter(t<10)
ggplot(a, aes(x = t, y= value)) +
          geom_line() +
          facet_wrap(~axis, dir = 'v', strip.position = 'right')

Let's get both the accelerometer and gyroscope data into the required format before we extract features from it using get_walk_features.

walk_accelerometer_data <- walk_data$userAcceleration
walk_accelerometer_data$t <- walk_data$timestamp

walk_gyroscope_data <- walk_data$rotationRate
walk_gyroscope_data$t <- walk_data$timestamp

Now let's get the default walk features.

walk_features <- get_walk_features(accelerometer_data = walk_accelerometer_data, 
                                   gyroscope_data = walk_gyroscope_data)
# Using the default settings

The output file is a list containing features extracted using default functions ($extracted_features), features calculated using models ($model_features) and an element for error ($error). Let's look at the $extracted_features to see how the default features look like

dplyr::glimpse(walk_features$extracted_features)

As with most of the functions in mhealthtools, get_walk_features also comes with many options for input parameters for versatile data analysis. Let's look at a few examples

Changing the max number of possible IMFs --

walk_features <- get_walk_features(
  accelerometer_data = accelerometer_data,
  gyroscope_data = gyroscope_data,
  IMF = 3)
# Changing the Max no. of possible IMFs to 3 (default is 2)

Applying bandpass frequency filter before running feature extraction pipeline --

walk_features <- get_walk_features(
  accelerometer_data = accelerometer_data,
  gyroscope_data = gyroscope_data,
  frequency_filter = c(3,15))
# Consider the frequencies only from 3Hz to 15Hz

Please read the function documentation ?get_walk_features for more info.

Tremor Module

Resting tremor

In this activity, the participant is asked to hold the phone in his/her hand in their lap so that their resting tremor can be measured. For someone with Parkinson's we expect to see some effects of the characteristic pill rolling tremor spread across gyroscope and accelerometer measurements.

The features extracted from get_balance_features do capture to some extent the effects of those symptoms.

mhealthtools comes with preloaded sample rest tremor data from the mPower study. Let's take a look at the data before we proceed to analyze it.

dplyr::glimpse(rest_tremor_data)

Let's look at the accelerometer data -- userAcceleration (plotting only the first ten seconds)

library(ggplot2)
a <- rest_tremor_data$userAcceleration
a$t <- rest_tremor_data$timestamp
a <- tidyr::gather(a, 'axis' , 'value', -t)
ggplot(a, aes(x = t, y= value)) +
          geom_line() +
          facet_wrap(~axis, dir = 'v', strip.position = 'right')

Let's look at the gyroscope data -- rotationRate (plotting only the first ten seconds)

library(ggplot2)
a <- rest_tremor_data$rotationRate
a$t <- rest_tremor_data$timestamp
a <- tidyr::gather(a, 'axis' , 'value', -t)
ggplot(a, aes(x = t, y= value)) +
          geom_line() +
          facet_wrap(~axis, dir = 'v', strip.position = 'right')

Let's get the accelerometer and gyroscope data into the required format before we extract features from it using get_balance_features.

rest_tremor_accelerometer_data <- rest_tremor_data$userAcceleration
rest_tremor_accelerometer_data$t <- rest_tremor_data$timestamp

rest_tremor_gyroscope_data <- rest_tremor_data$rotationRate
rest_tremor_gyroscope_data$t <- rest_tremor_data$timestamp

Now let's get the rest tremor features.

rest_tremor_features <- get_tremor_features(
  accelerometer_data = rest_tremor_accelerometer_data,
  gyroscope_data = rest_tremor_gyroscope_data)
# Using the default settings

The output file is a list similar to the get_walk_features described above. Let's take a look at the default features

dplyr::glimpse(rest_tremor_features$extracted_features)

As with most of the functions in mhealthtools, get_balance_features also comes with many options for input parameters for versatile data analysis

rest_tremor_features <- get_balance_features(
  accelerometer_data = rest_tremor_accelerometer_data,
  gyroscope_data = rest_tremor_gyroscope_data,
  frequency_filter = c(3,15),
  detrend = T,
  IMF = 4,
  time_filter = c(2,8),
  derived_kinematics = T,
  window_length = 256,
  window_overlap = 0.5)
# Consider the frequencies only from 3Hz to 15Hz
# detrend using Loess
# Max no. of IMFs = 4
# Consider time between 2 and 8sec
# derive kinematics for accelerometer and gyroscope. This would derive metrics like displacement, velocity, jerk and autocorrelation for accelerometer and gyrscope from their readings and then apply the feature extraction pipeline on each of these signals (i.e consider each derived metric as a independent input)
# window length is 256 samples
# 50% overlap between consecutive windows

Please read the function documentation ?get_tremor_features for more info.

Kinetic tremor (Action tremor)

In this activity, the participant is asked to hold the phone in their hand, and extend their hands to shoulder length and then touch their nose with their finger, and keep this motion in a loop.

The features extracted from get_kinetic_tremor_features do capture to some extent the effects of those symptoms. In the default processing, the signal is broken down into IMFs (Intrinsic Mode Functions) using the Hilbert Huang Transform before they are analyzed.

mhealthtools comes with preloaded sample kinetic tremor data from the mpower study. Let's take a look at the data before we proceed to analyze it.

dplyr::glimpse(kinetic_tremor_data)

Let's look at the accelerometer data -- userAcceleration (plotting only the first ten seconds)

library(ggplot2)
a <- kinetic_tremor_data$userAcceleration
a$t <- kinetic_tremor_data$timestamp
a <- tidyr::gather(a, 'axis' , 'value', -t)
a <- a %>% dplyr::filter(t<10)
ggplot(a, aes(x = t, y= value)) +
          geom_line() +
          facet_wrap(~axis, dir = 'v', strip.position = 'right')

Let's look at the gyroscope data -- rotationRate (plotting only the first ten seconds)

library(ggplot2)
a <- kinetic_tremor_data$rotationRate
a$t <- kinetic_tremor_data$timestamp
a <- tidyr::gather(a, 'axis' , 'value', -t)
a <- a %>% dplyr::filter(t<10)
ggplot(a, aes(x = t, y= value)) +
          geom_line() +
          facet_wrap(~axis, dir = 'v', strip.position = 'right')

Let's get the accelerometer and gyroscope data into the required format before we extract features from it using get_kinetic_tremor_features.

kinetic_tremor_accelerometer_data <- kinetic_tremor_data$userAcceleration
kinetic_tremor_accelerometer_data$t <- kinetic_tremor_data$timestamp

kinetic_tremor_gyroscope_data <- kinetic_tremor_data$rotationRate
kinetic_tremor_gyroscope_data$t <- kinetic_tremor_data$timestamp

Now let's get the kinetic tremor features.

kinetic_tremor_features <- get_kinetic_tremor_features(
  accelerometer_data = kinetic_tremor_accelerometer_data,
  gyroscope_data = kinetic_tremor_gyroscope_data)
# Using the default settings

The output file is a list similar to the get_walk_features described above. Let's take a look at the default features

dplyr::glimpse(kinetic_tremor_features$extracted_features)

As with most of the functions in mhealthtools, get_kinetic_tremor_features also comes with many options for input parameters for versatile data analysis

kinetic_tremor_features <- get_kinetic_tremor_features(
  accelerometer_data = rest_tremor_accelerometer_data,
  gyroscope_data = rest_tremor_gyroscope_data,
  frequency_filter = c(3,15),
  detrend = T,
  IMF = 4,
  time_filter = c(2,8),
  derived_kinematics = T,
  window_length = 256,
  window_overlap = 0.5)
# Consider the frequencies only from 3Hz to 15Hz
# detrend using Loess
# Max no. of IMFs = 4
# Consider time between 2 and 8sec
# derive kinematics for accelerometer and gyroscope. This would derive metrics like displacement, velocity, jerk and autocorrelation for accelerometer and gyrscope from their readings and then apply the feature extraction pipeline on each of these signals (i.e consider each derived metric as a independent input)
# window length is 256 samples
# 50% overlap between consecutive windows

Please read the function documentation ?get_kinetic_tremor_features for more info.

Tapping Module (Touchscreen)

In this activity, the participant is asked to tap the screen with their middle and pointing finger alternatingly as fast as they can for 30s (with one hand first, and if possible the other hand too). For people having Parkinson's they might not be able to tap as fast as a normal person, and also they might show higher fatigue (slower tap rate) at the end of the activity than a normal person.

Features extracted from get_tapping_features of mhealthtools try to address these symptoms.

mhealthtools comes with preloaded sample tapping data from the mpower study. Let's take a look at the data before we proceed to analyze it.

head(tap_data)

Let's look at a plot of the x and y co-ordinates of the tap data against t

library(ggplot2)
a <- tap_data
a <- tidyr::gather(a %>% dplyr::select(-buttonid), 'axis' , 'value', -t)
ggplot(a, aes(x = t, y= value)) + geom_line() + facet_wrap(~axis, dir = 'v', strip.position = 'right')

Extracting features using the default pipeline

tap_features <- get_tapping_features(tap_data)
# Using the default settings

dplyr::glimpse(tap_features)

As with most of the functions in mhealthtools, get_tapping_features also comes with options input parameters for versatile data analysis.

tap_features <- get_tapping_features(tap_data, depress_threshold = 10)
# Changing the threshold for intertap distance in x-axis, depress_threshold to 10 (default is 20)

Please read the function documentation ?get_tapping_features for more info.

Heartrate Module (Camera)

In this activity, the participant is asked to place their finger over their smartphone camera and from the average pixel intensity across frames, we estimate the heartrate of the participant.

To use the fucntion get_heartrate of mhealthtools the data needs to be in a certain format, fortunately the package comes preloaded with sampledata heartrate_data, let's take a look at it

head(heartrate_data)

Let's look at the mean centered red, blue and green columns of the heartrate data against t (plotting only the time between 2 and 10 seconds). Mean centering because otherwise it would be diffcult to compare the y-axis across plots.

library(ggplot2)
a <- heartrate_data %>% dplyr::select(t, red,blue,green)
a$red <- a$red - mean(a$red, na.rm = T)
a$green <- a$green - mean(a$green, na.rm = T)
a$blue <- a$blue - mean(a$blue, na.rm = T)
a <- a %>% dplyr::select(t, red_centered = red, green_centered = green, 
                         blue_centered = blue)
a <- tidyr::gather(a, 'axis' , 'value', -t)
a <- a %>% dplyr::filter(t<10, t>2)
ggplot(a, aes(x = t, y= value)) + geom_line()+facet_wrap(~axis)

Now to extract the heartrate, we just pass the formatted data into get_heartrate which outputs a list containing estimated heartrates for the three channels: red, blue and green, the sampling rate and an error status.

hr_values <- get_heartrate(heartrate_data)
# Using the default settings, a window of 10s, 50% overlap between consecutive windows,
# and autocorrelation based algorithm for the heartrate estimation and the appropriate
# pre-processing

The output is a list containing heartrate estimates from red, green and blue channels, and error and calculated sampling rate. Only the 'acf' method will output confidence values for the estimated heart rate.

dplyr::glimpse(hr_values)

As with most of the functions in mhealthtools, get_heartrate also comes with many options for input parameters for versatile data analysis.

hr_values <- get_heartrate(heartrate_data,
                           window_length = 8, 
                           window_overlap = 0.5,
                           method = 'peak')
# Using 8s windows, with a 50% overlap, and uses the autocorrelation method to find HR 
# Defaults are 10s windows, 50% overlap and acf method

The output is a list containing heartrate estimates from red, green and blue channels, and error and calculated sampling rate. The 'psd' (power spectral density) and 'peak' (peak picking based algorithm) don't have any confidence value associated with the estimated heart rate, so will output NA for confidence values

dplyr::glimpse(hr_values)

Please read the function documentation ?get_heartrate for more info.

Sage-Bionetworks/mhealthtools documentation built on Sept. 20, 2020, 1:42 p.m.