hutils
packageMy name is Hugh.
I've written some miscellaneous functions that don't seem to belong in a particular package.
I've usually put these in R/utils.R
when I write a package.
Thus, hutils
.
This vignette just goes through each exported function.
library(knitr) suggested_packages <- c("geosphere", "nycflights13", "dplyr", "ggplot2", "microbenchmark") opts_chunk$set(eval = all(vapply(suggested_packages, requireNamespace, quietly = TRUE, FUN.VALUE = FALSE)))
tryCatch({ library(geosphere) library(nycflights13) library(dplyr, warn.conflicts = FALSE) library(ggplot2) library(microbenchmark) library(data.table, warn.conflicts = FALSE) library(magrittr) library(hutils, warn.conflicts = FALSE) }, # requireNamespace does not detect errors like # package ‘dplyr’ was installed by an R version with different internals; it needs to be reinstalled for use with this R version error = function(e) { opts_chunk$set(eval = FALSE) }) options(digits = 4)
These are simple additions to magrittr
's aliases, including:
capitalized forms of and
and or
that invoke &&
and ||
(the 'long-form' logical operators)
and nor
/ neither
functions.
The main motivation is to make the source code easier to indent. I occasionally find such source code easier to use.
OR(OR(TRUE, stop("Never happens")), ## short-circuits AND(FALSE, stop("Never happens")))
nor
(or neither
which is identical) returns TRUE
if and only if both arguments are FALSE
.
coalesce
and if_else
These are near drop-in replacements for the equivalent functions from dplyr
.
They are included here because they are very useful outside of the tidyverse,
but may be required in circumstances where importing dplyr
(with all of its dependencies)
would be inappropriate.
They attempt to be drop-in replacements but:
hutils::if_else
only works with logical
, integer
, double
, and character
type vectors.
Lists and factors won't work.hutils::coalesce
short-circuits on its first argument; if there are no NA
s in x
then x
is returned, even
if the other vectors are the wrong length or type.In addition, hutils::if_else
is generally faster than dplyr::if_else
:
my_check <- function(values) { all(vapply(values[-1], function(x) identical(values[[1]], x), logical(1))) } set.seed(2) cnd <- sample(c(TRUE, FALSE, NA), size = 100e3, replace = TRUE) yes <- sample(letters, size = 100e3, replace = TRUE) no <- sample(letters, size = 100e3, replace = TRUE) na <- sample(letters, size = 100e3, replace = TRUE) microbenchmark(dplyr = dplyr::if_else(cnd, yes, no, na), hutils = hutils::if_else(cnd, yes, no, na), check = my_check) %>% print cnd <- sample(c(TRUE, FALSE, NA), size = 100e3, replace = TRUE) yes <- sample(letters, size = 1, replace = TRUE) no <- sample(letters, size = 100e3, replace = TRUE) na <- sample(letters, size = 1, replace = TRUE) microbenchmark(dplyr = dplyr::if_else(cnd, yes, no, na), hutils = hutils::if_else(cnd, yes, no, na), check = my_check) %>% print
This speed advantage also appears to be true of coalesce
:
x <- sample(c(letters, NA), size = 100e3, replace = TRUE) A <- sample(c(letters, NA), size = 100e3, replace = TRUE) B <- sample(c(letters, NA), size = 100e3, replace = TRUE) C <- sample(c(letters, NA), size = 100e3, replace = TRUE) microbenchmark(dplyr = dplyr::coalesce(x, A, B, C), hutils = hutils::coalesce(x, A, B, C), check = my_check) %>% print
especially during short-circuits:
x <- sample(c(letters), size = 100e3, replace = TRUE) microbenchmark(dplyr = dplyr::coalesce(x, A, B, C), hutils = hutils::coalesce(x, A, B, C), check = my_check) %>% print
x <- sample(c(letters, NA), size = 100e3, replace = TRUE) A <- sample(c(letters), size = 100e3, replace = TRUE) microbenchmark(dplyr = dplyr::coalesce(x, A, B, C), hutils = hutils::coalesce(x, A, B, C), check = my_check) %>% print
To drop a column from a data.table
, you set it to NULL
DT <- data.table(A = 1:5, B = 1:5, C = 1:5) DT[, A := NULL]
There's nothing wrong with this, but I've found the following a useful alias, especially in a magrittr
pipe.
DT <- data.table(A = 1:5, B = 1:5, C = 1:5) DT %>% drop_col("A") %>% drop_col("B") # or DT <- data.table(A = 1:5, B = 1:5, C = 1:5) DT %>% drop_cols(c("A", "B"))
These functions simple invoke the canonical form, so won't be any faster.
Additionally, one can drop columns by a regular expression using drop_colr
:
flights <- as.data.table(flights) flights %>% drop_colr("time") %>% drop_colr("arr(?!_delay)", perl = TRUE)
drop_constant_cols
When a table is filtered, the filtrate is often redundant.
flights %>% .[origin == "JFK"] %>% drop_constant_cols
drop_empty_cols
This function drops columns in which all the values are NA
.
planes %>% as.data.table %>% .[!complete.cases(.)] planes %>% as.data.table %>% .[!complete.cases(.)] %>% # drops speed drop_empty_cols
duplicated_rows
There are many useful functions for detecting duplicates in R.
However, in interactive use, I often want to not merely see which values are duplicated,
but also compare them to the original.
This is especially true when I am comparing duplicates across a subset of columns in a a data.table
.
flights %>% # only the 'second' of the duplicates is returned .[duplicated(., by = c("origin", "dest"))] flights %>% # Both rows are returned and (by default) # duplicates are presented adjacently duplicated_rows(by = c("origin", "dest"))
To emphasize the miscellany of this package, I now present haversine_distance
which simply returns the distance between two points on the Earth,
given their latitude and longitude.
I prefer this to other packages' implementations.
Although the geosphere
package can do a lot more than calculate distances between points, I find the interface for distHaversine
unfortunate as it cannot be easily used inside a data.frame
. In addition, I've found the arguments clearer in hutils::haversine_distance
rather than trying to remember whether to use byrow
inside the matrix
function while passing to distHaversine
.
DT1 <- data.table(lat_orig = runif(1e5, -80, 80), lon_orig = runif(1e5, -179, 179), lat_dest = runif(1e5, -80, 80), lon_dest = runif(1e5, -179, 179)) DT2 <- copy(DT1) microbenchmark(DT1[, distance := haversine_distance(lat_orig, lon_orig, lat_dest, lon_dest)], DT2[, distance := distHaversine(cbind(lon_orig, lat_orig), cbind(lon_orig, lat_orig))]) rm(DT1, DT2)
mutate_other
There may be occasions where a categorical variable in a data.table
may need
to modified to reduce the number of distinct categories. For example, you may
want to plot a chart with a set number of facets, or ensure the smooth operation of
randomForest
, which accepts no more than
32 levels in a feature.
mutate_other
keeps the n most common categories and changes the
other categories to Other
.
set.seed(1) DT <- data.table(Fruit = sample(c("apple", "pear", "orange", "tomato", "eggplant"), size = 20, prob = c(0.45, 0.25, 0.15, 0.1, 0.05), replace = TRUE), Price = rpois(20, 10)) kable(mutate_other(DT, "Fruit", n = 3)[])
ngrep
This is a 'dumb' negation of grep
.
In recent versions of R, the option invert = FALSE
exists.
A slight advantage of ngrep
is that it's shorter to type.
But if you don't have arthritis, best use invert = FALSE
or !grepl
.
notin
ein
enotin
pin
These functions provide complementary functionality to %in%
:
%notin%
%notin%
is the negation of %in%
, but also uses the package fastmatch
to increase the speed of the operation
%ein%
and %enotin%
The functions %ein%
and %enotin%
are motivated by a different sort of problem.
Consider the following statement:
iris <- as.data.table(iris) iris[Species %in% c("setosa", "versicolour")] %$% mean(Sepal.Length / Sepal.Width)
On the face of it, this appears to give the average ratio of Iris setosa and Iris versicolour irises. However, it only gives the average ratio of setosa irises, as the correct spelling is Iris versicolor not -our. This particular error is easy to make, (in fact when I wrote this vignette, the first hit of Google for iris dataset
made the same spelling error), but it's easy to imagine similar mistakes, such as mistaking the capitalization of a value.
The functions %ein%
and %enotin%
strive to reduce the occurrence of this mistake.
The functions operate exactly the same as %in%
and %enotin%
but error if any of the table of values to be matched against is not present in any of the values:
iris <- as.data.table(iris) iris[Species %ein% c("setosa", "versicolour")] %$% mean(Sepal.Length / Sepal.Width)
The e
stands for 'exists'; i.e. they should be read as "exists and in" and "exists and not in".
%pin%
This performs a partial match (i.e grepl
) but with a possibly more readable or intuitive syntax
identical(iris[grep("v", Species)], iris[Species %pin% "v"])
If the RHS has more than one element, the matching is done on alternation (i.e. OR):
iris[Species %pin% c("ver", "vir")] %>% head
There is an important qualification: if the RHS is NULL
, then the result will be TRUE
along the length of x
, contrary to the behaviour of %in%
.
This is not entirely unexpected as NULL
could legitimately be interpreted as (\varepsilon), the empty regular expression, which occurs in every string.
provide.dir
This is the same as dir.create
but checks whether the target exists or not and does nothing if it does. Motivated by \providecommand
in (\rm\LaTeX{}), which creates a macro only if it does not exist already.
select_which
This provides a similar role to dplyr::select_if
but was originally part of package:grattan
so has a different name. It simply returns the columns whose values return TRUE
when Which
is applied.
Additional columns (which may or not may satisfy Which
) may be included by using .and.dots
.
(To remove columns, you can use drop_col
).
DT <- data.table(x = 1:5, y = letters[1:5], AB = c(NA, TRUE, FALSE, TRUE, FALSE)) select_which(DT, anyNA, .and.dots = "y")
set_cols_first
Up to and including data.table 1.10.4
, one could only reorder the columns by supplying all the columns.
You can use set_cols_first
and set_cols_last
to put columns first or last without supplying all the columns.
In some circumstances, you need to know that the key
of a data.table
is unique.
For example, you may expect a join to be performed later, without specifying mult='first'
or permitting Cartesian joins.
data.table
does not require a key
to be unique and does not supply tools to check the uniqueness of keys. hutils
supplies two simple functions: has_unique_key
which when applied to a data.table
returns TRUE
if and only if the table has a key and it is unique.
set_unique_key
does the same as setkey
but will error if the resultant key is not unique.
hutils
v1.1.0auc
The area under the (ROC) curve gives a single value to measure the tradeoff between true positives and false positives.
dt <- data.table(y = !sample(0:1, size = 100, replace = TRUE), x = runif(100)) dt[, pred := predict(lm(y ~ x, data = .SD), newdata = .SD)] dt[, auc(y, pred)]
select_grep
To select columns matching a regular expression:
flights %>% select_grep("arr")
You can use the additional arguments .and
and .but.not
to override the patterns.
flights %>% select_grep("arr", .and = "year", .but.not = "arr_time")
hutils
v1.2.0RQ
This is simply a shorthand to test whether a package needs installing. The package name need not be quoted, for convenience.
RQ(dplyr, "dplyr must be installed") RQ("dplyr", "dplyr needs installing", "dplyr installed.")
ahull
This locates the biggest rectangle beneath a curve:
if (!identical(Sys.info()[["sysname"]], "Darwin")) ggplot(data.table(x = c(0, 1, 2, 3, 4), y = c(0, 1, 2, 0.1, 0))) + geom_area(aes(x, y)) + geom_rect(data = ahull(, c(0, 1, 2, 3, 4), c(0, 1, 2, 0.1, 0)), aes(xmin = xmin, xmax = xmax, ymin = ymin, ymax = ymax), color = "red")
set.seed(101) ahull_dt <- data.table(x = c(0:100) / 100, y = cumsum(rnorm(101, 0.05))) if (!identical(Sys.info()[["sysname"]], "Darwin")) ggplot(ahull_dt) + geom_area(aes(x, y)) + geom_rect(data = ahull(ahull_dt), aes(xmin = xmin, xmax = xmax, ymin = ymin, ymax = ymax), color = "red") + geom_rect(data = ahull(ahull_dt, incl_negative = TRUE), aes(xmin = xmin, xmax = xmax, ymin = ymin, ymax = ymax), color = "blue") + geom_rect(data = ahull(ahull_dt, incl_negative = TRUE, minH = 4), aes(xmin = xmin, xmax = xmax, ymin = ymin, ymax = ymax), color = "green") + geom_rect(data = ahull(ahull_dt, incl_negative = TRUE, minW = 0.25), aes(xmin = xmin, xmax = xmax, ymin = ymin, ymax = ymax), color = "white", fill = NA)
hutils
v1.3.0weighted_quantile
Simply a version of quantile
supporting weighted values:
x <- 1:10 w <- c(rep(1, 5), rep(2, 5)) quantile(x, prob = c(0.25, 0.75), names = FALSE) weighted_quantile(x, w, p = c(0.25, 0.75))
mutate_ntile
To add a column of ntiles (say, for later summarizing):
flights %>% as.data.table %>% .[, .(year, month, day, origin, dest, distance)] %>% mutate_ntile(distance, n = 5L)
You can use non-standard evaluation (as above) or you can quote the col
argument. Use character.only = TRUE
to ensure column is only interpreted
as character.
flights %>% as.data.table %>% .[, .(year, month, day, origin, dest, distance)] %>% mutate_ntile(distance, n = 5L)
flights %>% as.data.table %>% mutate_ntile("distance", n = 5L, character.only = TRUE) %>% .[, dep_delay := coalesce(dep_delay, 0)] %>% .[, .(avgDelay = mean(dep_delay)), keyby = "distanceQuintile"]
longest_affix
Trim common affixes can be useful during data cleaning:
trim_common_affixes(c("CurrentHousingCosts(weekly)", "CurrentFuelCosts(weekly)"))
hutils 1.4.0
%<->%
Referred to as swap
in the documentation. Used to swap values between object names
a <- 1 b <- 2 a %<->% b identical(c(a, b), c(2, 1))
average_bearing
Determine the average bearing of vectors. Slightly more difficult than simply the average modulo 360 since its the most acute sector is desired.
average_bearing(0, 270) # NW mean(c(0, 270)) # SE (i.e. wrong)
dir2
This is a faster version of list.files
for Windows only, utilizing the dir
command
on the command prompt.
Mode
The statistical mode; the most common element.
Mode(c(1, 1, 1, 2, 3))
replace_pattern_in
A cousin of find_pattern_in
, but instead of collecting the results, it replaces the
contents sought with the replacement provided.
samp
A safer version of sample
. I use it because I found the following behaviour of sample
surprising.
DT <- data.table(x = c(5, 2, 3), y = c(5, 3, 4)) DT[, .(Base = sample(.BY[["x"]]:.BY[["y"]])), keyby = .(x, y)] DT[, .(Base = samp(.BY[["x"]]:.BY[["y"]])), keyby = .(x, y)]
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.