Persistent Indexes and Lifecycle

In bigANNOY: Approximate k-Nearest Neighbour Search for 'bigmemory' Matrices with Annoy

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

options(bigANNOY.progress = FALSE)
set.seed(20260326)

bigANNOY v3 adds explicit index lifecycle support around persisted Annoy files. That makes it possible to:

• build an index once and reopen it later
• choose whether a reopened index should load eagerly or lazily
• check whether a native handle is currently live
• close loaded handles explicitly in long sessions
• validate the Annoy file against its recorded metadata before reuse

This vignette focuses on those operational workflows rather than on search quality or benchmark tuning.

Why Lifecycle Management Matters

Annoy indexes are stored on disk. In practice, that means the useful object is not just the result of a single build call, but a persisted pair:

• the .ann index file
• the .meta sidecar metadata file

The bigannoy_index object returned by bigANNOY is a session-level wrapper around those files. It remembers the key metadata and can optionally hold a live native handle for faster repeated searches within the same R session.

Load the Packages

library(bigANNOY)
library(bigmemory)

Build an Index in Lazy Mode

We will create a small reference matrix, write the Annoy index into a temporary directory, and keep the returned object in lazy mode so the first search is what loads the live handle.

artifact_dir <- file.path(tempdir(), "bigannoy-lifecycle")
dir.create(artifact_dir, recursive = TRUE, showWarnings = FALSE)

ref_dense <- matrix(
  c(
    0.0, 0.1, 0.2,
    0.1, 0.0, 0.1,
    0.2, 0.1, 0.0,
    1.0, 1.1, 1.2,
    1.1, 1.0, 1.1,
    1.2, 1.1, 1.0
  ),
  ncol = 3,
  byrow = TRUE
)

ref_big <- as.big.matrix(ref_dense)
index_path <- file.path(artifact_dir, "ref.ann")
metadata_path <- paste0(index_path, ".meta")

index <- annoy_build_bigmatrix(
  ref_big,
  path = index_path,
  n_trees = 25L,
  metric = "euclidean",
  seed = 123L,
  load_mode = "lazy"
)

index

The returned object points to the persisted files, but the native handle is not loaded yet.

annoy_is_loaded(index)
file.exists(index$path)
file.exists(index$metadata_path)

Inspect the Sidecar Metadata

The sidecar metadata file is meant to support safe reopen and validation workflows. It records the metric, dimension, item count, build settings, and a small file signature for the persisted Annoy file.

metadata <- read.dcf(index$metadata_path)
metadata[, c(
  "index_id",
  "metric",
  "n_dim",
  "n_ref",
  "n_trees",
  "build_seed",
  "build_backend",
  "file_size",
  "file_md5"
)]

The important point is not the exact formatting of the metadata file, but that the persisted index is now self-describing enough to be reopened and checked in later sessions.

Lazy Loading Versus Eager Loading

There are two lifecycle modes:

• "lazy" keeps only metadata in memory until the first search
• "eager" loads a native handle immediately when the index object is created or reopened

The index we just built is lazy.

annoy_is_loaded(index)

The first search loads the handle automatically.

first_result <- annoy_search_bigmatrix(index, k = 2L, search_k = 100L)

annoy_is_loaded(index)
first_result$index
round(first_result$distance, 3)

Once the handle is loaded, repeated searches in the same session can reuse it.

second_result <- annoy_search_bigmatrix(index, k = 2L, search_k = 100L)

identical(first_result$index, second_result$index)
all.equal(first_result$distance, second_result$distance)

Validate Without Loading

Validation and loading are related, but they are not the same thing. Sometimes you want to confirm that the metadata and file signature still look right without paying the cost of loading the native handle yet.

annoy_close_index(index)
annoy_is_loaded(index)

validation_no_load <- annoy_validate_index(
  index,
  strict = TRUE,
  load = FALSE
)

validation_no_load$valid
validation_no_load$checks[, c("check", "passed", "severity")]
annoy_is_loaded(index)

Because load = FALSE, the validation report checks the recorded metadata against the current file without changing the loaded state of the object.

Validate and Load Explicitly

If you do want validation to also confirm that the Annoy index can be opened successfully, set load = TRUE.

validation_with_load <- annoy_validate_index(
  index,
  strict = TRUE,
  load = TRUE
)

validation_with_load$valid
tail(validation_with_load$checks[, c("check", "passed", "severity")], 2L)
annoy_is_loaded(index)

This is a useful pattern before long-running queries or before handing a reopened index to downstream analysis code.

Close a Loaded Handle Explicitly

Explicit close support is helpful in long R sessions, in tests, and in code that wants deterministic control over when handles are released.

annoy_close_index(index)
annoy_is_loaded(index)

The persisted .ann file is still there, so the next search can load it again.

reload_result <- annoy_search_bigmatrix(index, k = 2L, search_k = 100L)

annoy_is_loaded(index)
reload_result$index

Reopen the Same Index in a New Object

The more important persistence workflow is reopening the same files into a new bigannoy_index object. This is what a later R session would typically do.

annoy_open_index() and annoy_load_bigmatrix() both support this pattern. The main distinction is semantic: annoy_load_bigmatrix() is a friendlier name when you are thinking in terms of bigmemory workflows, while annoy_open_index() makes the persisted-index lifecycle more explicit.

reopened_lazy <- annoy_open_index(
  path = index$path,
  load_mode = "lazy"
)

reopened_eager <- annoy_load_bigmatrix(
  path = index$path,
  load_mode = "eager"
)

annoy_is_loaded(reopened_lazy)
annoy_is_loaded(reopened_eager)

The eager reopen path loads immediately. The lazy reopen path waits until first use.

reopened_result <- annoy_search_bigmatrix(
  reopened_lazy,
  k = 2L,
  search_k = 100L
)

annoy_is_loaded(reopened_lazy)
reopened_result$index

Lifecycle State Lives in the Session Object

The persisted files are shared, but loaded-state tracking is per-object and per-session. Closing one in-memory object does not invalidate another object that already opened the same index.

annoy_close_index(reopened_lazy)
c(
  original = annoy_is_loaded(index),
  reopened_lazy = annoy_is_loaded(reopened_lazy),
  reopened_eager = annoy_is_loaded(reopened_eager)
)

This is a useful mental model:

• the .ann file is the durable asset
• the bigannoy_index object is the session-level controller
• the loaded handle is cached inside that controller only for the current session

What Happens If Validation Fails?

In normal workflows, annoy_validate_index(..., strict = TRUE) is the safest default because it stops immediately when critical checks fail. If you want a diagnostic report instead of an error, use strict = FALSE.

report <- annoy_validate_index(
  reopened_eager,
  strict = FALSE,
  load = FALSE
)

report$valid
report$checks[, c("check", "passed", "severity")]

That pattern is especially helpful when you are writing higher-level code that wants to display a validation report before deciding whether to rebuild or reload an index.

Recommended Workflow

For most projects, a sensible lifecycle pattern looks like this:

1. build the index once with annoy_build_bigmatrix()
2. keep the .ann file and the .meta file together
3. reopen with annoy_open_index() or annoy_load_bigmatrix() in later sessions
4. run annoy_validate_index() before important downstream work
5. use lazy loading for lighter startup or eager loading for repeated search sessions
6. call annoy_close_index() when you want explicit control over loaded handles

Recap

bigANNOY v3 turns persisted Annoy files into a more explicit lifecycle:

• build once, reopen later
• choose eager or lazy loading
• test loaded state with annoy_is_loaded()
• close handles with annoy_close_index()
• validate persisted files with annoy_validate_index()

The next vignette to read after this one is usually File-Backed bigmemory Workflows, which focuses on descriptor files, file-backed matrices, and streamed output destinations.

bigANNOY documentation built on April 1, 2026, 9:07 a.m.