Custom Pattern Search with dig()

In nuggets: Fast and Extensible Pattern Discovery in Tabular Data

knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>"
)

options(tibble.width = Inf)

Introduction

dig() is the general function behind custom pattern search in the nuggets package. It searches for patterns of a custom type by generating conditions as elementary conjunctions of predicates and by executing a user-defined callback function on each generated condition.

This makes dig() the low-level building block behind more specialized functions such as dig_associations() and dig_correlations(). Use it when you want to keep the search over conditions, but define your own evaluation logic.

This vignette focuses on how to use the dig() function: how to write the callback function and how to control the search. For preparation of crisp and fuzzy predicates, see vignette("data-preparation").

Examples in this vignette require loading the following packages:

library(nuggets)
library(dplyr)     # for data manipulation

A Small Working Dataset

dig() expects a matrix or data frame whose columns are logical predicates or numeric fuzzy predicates. We will use iris and prepare a small crisp predicate dataset that is rich enough for the first examples below:

crisp_iris <- iris |>
    partition(Species) |>
    partition(Sepal.Length:Petal.Width, .method = "crisp", .breaks = 3)

head(crisp_iris, n = 3)

The commands above create crisp predicates for the four numeric columns of iris, plus the three species predicates. The preparation step is intentionally brief here; the dedicated vignette("data-preparation") explains partition(), fuzzy predicates, and breakpoints in detail.

A Simple dig() Call

The dig() function generates conditions from the selected predicates in a recursive manner. It starts with the empty condition and adds one predicate at a time, up to the specified max_length. Meanwhile, it evaluates the generated condition and tests whether it meets the minimum support requirement. By support we mean the relative frequency of rows satisfying the condition. If the condition is frequent enough, dig() calls the user-defined callback function with the generated condition and other information. The callback can then compute any output you want. The dig() function collects those outputs and returns them as a list.

The simplest callback function can handle just the generated condition. In the following example, the callback generates some debug output and returns the formatted condition:

simple_callback <- function(condition) {
    str(condition)
    cat("------\n")

    list(condition = format_condition(names(condition)))
}

simple_result <- dig(x = crisp_iris,
                     f = simple_callback,
                     condition = starts_with("Sepal"),
                     min_length = 0,
                     max_length = 2,
                     min_support = 0.2)

As you can see from the debug output issued by the str() call within the callback, dig() enumerates all conditions that can be formed from the selected predicates (in this case, all predicates starting with "Sepal") and that meet the minimum support requirement. The callback receives each condition in the form of a named integer vector, where the names are the predicate names and the values are the column indices in the original data frame. The callback then returns a named list with the formatted condition. All callback results are collected into a list and returned by dig():

str(simple_result)

As you can see, the result is a list of named lists, one for each condition that was generated and passed to the callback. You can flatten the result into a tibble with dplyr's bind_rows():

bind_rows(simple_result)

Note also the attributes of the result list. They contain information about the search, such as the search statistics and the arguments that were passed to dig(). You can use this information for debugging or for reproducing the search later. For example, you can obtain the vector of predicate names that were used to generate conditions with:

attributes(simple_result)$call_args$condition

This simple example illustrates the basic workflow:

1. choose which predicates may form conditions;
2. set the search parameters (length, support, etc.);
3. define a callback function that computes the desired output for each condition;
4. let dig() enumerate conditions and collect callback results.

Condition and Focus

The simple example above only used the predicates for generating conditions. In many cases, you will also want to evaluate other predicates within each generated condition. For example, you may want to know how often each species occurs within a condition. That's where the foci (plural of focus) come into play.

Condition and focus predicates are selected separately with the condition and focus arguments of dig(). These arguments accept tidyselect expressions for selecting columns of the input data frame x.

Using Foci

Foci are predicates that are not used to generate conditions, but are evaluated within each generated condition. You can select foci with the focus argument of dig(). The callback function can then receive information about how often each focus occurs within the generated condition. This is useful, e.g., for finding conditions that are strongly associated with certain foci.

For instance, let us define a callback that provides the number of occurences of each species within each generated condition:

focus_callback <- function(condition, sum, pp) { 
    str(list(condition = condition,
             sum = sum,
             species = pp))
    cat("------\n")

    NULL
}

focus_result <- dig(x = crisp_iris,
                    f = focus_callback,
                    condition = starts_with("Sepal"),
                    focus = starts_with("Species"),
                    min_length = 2,
                    max_length = 2,
                    max_results = 1)

We have defined a callback that, besides condition, also receives sum and pp. The sum argument provides the number of rows satisfying the generated condition, and the pp argument contains the number of rows that satisfy both the generated condition and each focus predicate. In this case, the focus predicates are the species predicates, so pp tells us how many rows of each species satisfy the generated condition.

Also note that we are using a neat trick that is useful during development of the callback: we set max_results = 1 to stop the search after the first condition that meets the criteria. This allows us to see the output of the callback without waiting for the entire search to complete, which can be time-consuming for large datasets or complex conditions.

So far, our callback only prints the information to the console and returns NULL. Once we understand the structure of the data we receive, we can modify the callback to return a list of patterns, where each pattern contains the formatted condition, a single species, the count of data rows satisfying the condition, and the count of the species within that condition:

focus_callback <- function(condition, sum, pp) { 
    species_names <- names(pp)
    species_counts <- as.integer(pp)

    lapply(seq_along(species_names), function(i) {
        list(condition = format_condition(names(condition)),
             species = species_names[i],
             condition_count = sum,
             species_count = species_counts[i])
    })
}

focus_result <- dig(x = crisp_iris,
                    f = focus_callback,
                    condition = starts_with("Sepal"),
                    focus = starts_with("Species"),
                    min_length = 0,
                    max_length = 2) 

The result of dig() is a list of lists, where each inner list corresponds to a species within a generated condition. We use unlist(recursive = FALSE) to flatten the list of lists into a single list of patterns, and then bind_rows() to convert it into a tibble for easier viewing:

focus_result |>
    unlist(recursive = FALSE) |>
    bind_rows() |>
    head(n = 6)

Filtering Foci

As discussed in the previous section, dig() evaluates the focus predicates within each generated condition. You may or may not want to keep all foci for each condition. Some patterns may require all foci to be evaluated every time, while others may only require a subset of foci to be considered that are sufficiently frequent within the condition. Therefore, dig() provides several arguments to filter foci based on their support:

• min_focus_support: minimum support of a focus within a condition. I.e., the relative frequency of rows satisfying both the condition and the focus must be at least this value for the focus to be kept. Foci with support below this threshold are filtered out.
• min_conditional_focus_support: minimum conditional support of a focus within a condition. I.e., the relative frequency of rows satisfying both the condition and the focus, divided by the number of rows satisfying the condition, must be at least this value for the focus to be kept. Foci with conditional support below this threshold are filtered out.

The focus filtering may result in some conditions having no remaining foci. If you want to skip the callback for such conditions, set filter_empty_foci = TRUE. Otherwise, the callback will be called with empty focus information. Filtering empty foci also improves performance, because it also stops early the evaluation of longer conditions that would not have any remaining foci anyway.

The following example shows how this can be used to emulate association-rule search with a custom callback. Association rules are implications of the form "if condition then focus". Condition is named the antecedent and focus is named the consequent. The callback computes the confidence of each antecedent-consequent pair, filters the pairs based on minimum support and confidence, and returns a list of rules:

min_support <- 0.1
min_confidence <- 0.8

rule_callback <- function(condition, pp, support) {
    conf <- pp / support / nrow(crisp_iris)
    sel <- !is.na(conf) & conf >= min_confidence & !is.na(pp) & pp >= min_support
    conf <- conf[sel]
    supp <- pp[sel] / nrow(crisp_iris)

    lapply(seq_along(conf), function(i) {
        list(antecedent = format_condition(names(condition)),
             consequent = names(conf)[[i]],
             antecedent_support = support,
             rule_support = supp[[i]],
             confidence = conf[[i]]
        )
    })
}

rule_result <- dig(x = crisp_iris,
                   f = rule_callback,
                   condition = !starts_with("Species"),
                   focus = starts_with("Species"),
                   min_length = 1,
                   min_support = min_support,
                   min_focus_support = min_support,
                   min_conditional_focus_support = min_confidence,
                   filter_empty_foci = TRUE) |>
    unlist(recursive = FALSE) |>
    bind_rows() |>
    arrange(desc(confidence))

head(rule_result, n = 6)

This is a useful illustration of focus filtering, but association rules already have a dedicated implementation: dig_associations() searches for them more efficiently. For that purpose, prefer dig_associations() and see vignette("association-rules").

What the Callback Function Can Receive

As seen in the previous section, the callback function f may obtain not only the generated condition, but also other information. The amount of received information is controlled by declaring the arguments of the callback function. dig() inspects the callback function argument names and computes only the requested values. This is important for performance, because some values are expensive to compute and may not be needed for every search.

The callback function may declare any subset of the following arguments:

• condition: named integer vector of column indices representing the generated condition.
• sum: number of rows satisfying the condition for logical data, or the sum of truth degrees for fuzzy data.
• support: relative frequency of the condition, i.e., sum / nrow(x).
• indices: row indices of the original dataset x satisfying the condition in crisp searches or the indices of rows with non-zero truth degrees in fuzzy searches.
• weights: per-row truth degrees of the condition in dataset x; logical (crisp) data is treated as 0/1 weights.
• pp, pn, np, nn: contingency-table entries for foci. The i-th entry of each vector corresponds to the i*-th focus predicate. The entries are defined as follows:
• pp: sum of truth degrees of rows satisfying both the condition and the focus (positive condition, positive focus),
• pn: sum of truth degrees of rows satisfying the condition but not the focus, (positive condition, negative focus),
• np: sum of truth degrees of rows satisfying the focus but not the condition, (negative condition, positive focus),
• nn: sum of truth degrees of rows satisfying neither (negative condition, negative focus).

In practice:

• use condition when you need condition predicate names,
• use support or sum for condition-level filtering or ranking,
• use indices when you want to compute something on the original rows,
• use weights for custom fuzzy summaries,
• use pp, pn, np, and nn when your pattern depends on foci and their frequency within the condition.

Note: only declare the arguments you need. For example, if you don't need foci, don't declare pp, pn, np, or nn. This will save computation time, especially for large datasets or complex conditions. The most expensive values to compute are indices and weights. They require scanning the entire dataset for each generated condition. So avoid them if not needed.

Advanced Examples

Example: Fixed-Variable Correlations

dig_correlations() searches over both generated conditions and combinations of numeric variables. A simpler custom variant can be built directly with dig() when the two variables are fixed in advance and only the condition should vary.

Here we search for conditions under which Sepal.Length and Petal.Length correlate strongly. The callback receives indices, uses them to select the corresponding rows from the original iris data, and runs cor.test() on that sub-data:

correlation_callback <- function(condition, support, indices) {
    if (length(indices) < 10) {
        return(NULL)
    }
    fit <- cor.test(iris$Sepal.Length[indices],
                    iris$Petal.Length[indices],
                    method = "pearson")

    list(condition = format_condition(names(condition)),
         support = support,
         correlation = unname(fit$estimate),
         p_value = fit$p.value,
         n = length(indices))
}

correlation_result <- dig(x = crisp_iris,
                          f = correlation_callback,
                          condition = everything(),
                          min_length = 1,
                          max_length = 2,
                          min_support = 0.1) |>
    bind_rows() |>
    arrange(desc(abs(correlation)))

head(correlation_result, n = 6)

This example follows the same idea as dig_correlations():

• generate conditions,
• evaluate a statistic on the sub-data induced by each condition,
• return one row per successful evaluation.

The difference is that dig() leaves the statistic entirely in your hands. That is useful when you want to fix the variables, apply a custom test, return additional diagnostics, or combine several criteria in one callback.

Example: Handling Fuzzy Data

For fuzzy searches, conditions are no longer simply satisfied or not satisfied. Instead, each row has a truth degree in the interval $[0,1]$. In that setting, indices and weights play different roles:

• indices tell you which rows have a non-zero truth degree for the condition,
• weights tell you on scale $[0, 1]$ how strongly each row satisfies the condition.

The following example prepares fuzzy predicates from iris and then compares an unweighted summary based on indices with a weighted summary based on weights:

fuzzy_iris <- iris |>
    partition(Species) |>
    partition(Sepal.Length:Petal.Width, .method = "triangle", .breaks = 3)

head(fuzzy_iris, n = 3)

fuzzy_callback <- function(condition, indices, weights) {
    if (length(indices) < 20) {
        return(NULL)
    }

    list(condition = format_condition(names(condition)),
         nonzero_rows = sum(indices),
         weighted_support = sum(weights) / nrow(fuzzy_iris),
         mean_petal_length_by_indices = mean(iris$Petal.Length[indices]),
         mean_petal_length_by_weights = weighted.mean(iris$Petal.Length, weights))
}

fuzzy_result <- dig(x = fuzzy_iris,
                    f = fuzzy_callback,
                    condition = starts_with("Sepal"),
                    min_length = 1,
                    max_length = 1,
                    min_support = 0.2) |>
    bind_rows()

fuzzy_result

The unweighted mean based on indices treats all rows with non-zero membership equally. The weighted mean based on weights respects the fuzzy truth degrees, so rows that satisfy the condition more strongly contribute more. This is the main practical difference: indices identify the relevant rows, while weights quantify the strength of their membership.

Practical Notes

• Condition predicates are used to generate conditions, while focus predicates are evaluated within each generated condition. Use condition and focus arguments to select them separately.
• The result of dig() is a list. When each callback returns one named list, bind_rows() is a convenient way to flatten it into a tibble.
• If the callback returns multiple patterns per condition, return a list of named lists and flatten the result afterwards.
• Use only the arguments you need in the callback function. This saves computation time, especially for large datasets or complex conditions.
• For fuzzy searches, use weights when truth degrees matter, and indices when you only need the rows with non-zero membership.

Summary

dig() is the most flexible search interface in nuggets. It lets you:

1. generate conditions from selected predicates,
2. optionally evaluate focus predicates within each condition,
3. receive only the callback inputs you need,
4. define your own pattern logic, statistics, and output format.

Use dig() when the built-in search functions are close to what you need, but not exact. For related material, see:

• vignette("data-preparation") for creating crisp and fuzzy predicates from raw data,
• vignette("association-rules") for searching for association rules with dig_associations(),
• vignette("nuggets") for an overview of the package and its main workflows.

nuggets documentation built on Aug. 20, 2026, 5:07 p.m.