tidymodels Integration

In foundryR: Data Frame Workflows for 'Microsoft Foundry'

fixture_dir <- "tidymodels"
recording <- nzchar(Sys.getenv("FOUNDRY_RECORD_DOCS"))
have_fixtures <- dir.exists(fixture_dir) && length(list.files(fixture_dir)) > 0
run_api <- requireNamespace("httptest2", quietly = TRUE) &&
  (recording || have_fixtures)
have_tidymodels <- requireNamespace("tidymodels", quietly = TRUE)

# Attach foundryR before start_vignette(): httptest2 only sources the package's
# inst/httptest2/start-vignette.R (which sets replay placeholders) from attached
# packages.
library(foundryR)

if (run_api) {
  httptest2::start_vignette(fixture_dir)
}

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = run_api && have_tidymodels
)

Introduction

foundryR integrates with tidymodels through step_foundry_embed(), a recipe step that converts text columns into embedding vectors. Use it when text should enter a model as numeric predictors rather than as bag-of-words counts.

Why use embeddings in ML pipelines?

Traditional text features like bag-of-words or TF-IDF capture word frequencies but miss semantic meaning. Embeddings provide dense vector representations that understand:

• Synonyms: "happy" and "joyful" produce similar vectors
• Context: "bank" (financial) vs "bank" (river) are distinguished
• Relationships: Semantic similarities are preserved in vector space

By converting text to embeddings within a recipe, you get:

• Reproducible preprocessing: The embedding step is part of your documented workflow
• Consistent handling: Training and test data are processed identically
• Pipeline integration: Combine embeddings with other preprocessing steps

Prerequisites

Install tidymodels if you haven't already:

install.packages("tidymodels")

Ensure foundryR is configured with your Azure credentials and you have an embedding model deployed.

Basic Usage

Creating a Recipe with Embeddings

Use step_foundry_embed() to add embedding generation to your recipe. Creating the recipe is local and runs when the suggested tidymodels package is installed; prep() and bake() need the recorded API fixtures when rendering:

library(tidymodels)
library(foundryR)

reviews <- tibble(
  text = c(
    "This product is useful and easy to use.",
    "The setup was confusing and slow.",
    "The examples were clear and helpful.",
    "I needed better instructions."
  ),
  sentiment = factor(c("positive", "negative", "positive", "negative"))
)


recipe_spec <- recipe(sentiment ~ text, data = reviews) %>%
  step_foundry_embed(
    text,
    model = "text-embedding-3-small",
    keep_original = FALSE
  )

recipe_spec

Preparing and Baking the Recipe

prepped_recipe <- prep(recipe_spec, training = reviews)

baked_data <- bake(prepped_recipe, new_data = NULL)
baked_data

The text column is replaced with 1,536 numeric embedding dimensions (the exact number depends on your embedding model).

Complete ML Pipeline Example

Here's a full example building a sentiment classifier. It is shown as code only because fitting and resampling would repeat embedding API calls:

library(tidymodels)
library(foundryR)

# Load your data
set.seed(123)
reviews <- tibble(
  review_text = c(
    # Positive reviews
    "Absolutely love this product! Works perfectly.",
    "Great quality and fast shipping. Very satisfied.",
    "Best purchase I've made this year. Highly recommend!",
    "Exceeded all expectations. Will buy again.",
    "Perfect fit and great value for money.",
    # Negative reviews
    "Complete waste of money. Broke after one use.",
    "Terrible customer service. Never buying again.",
    "Poor quality, doesn't work as advertised.",
    "Disappointed. Much smaller than expected.",
    "Arrived damaged and took forever to ship."
  ),
  sentiment = factor(rep(c("positive", "negative"), each = 5))
)

# Split data
splits <- initial_split(reviews, prop = 0.8, strata = sentiment)
train_data <- training(splits)
test_data <- testing(splits)

# Define recipe with embeddings
embedding_recipe <- recipe(sentiment ~ review_text, data = train_data) %>%
  step_foundry_embed(
    review_text,
    model = "text-embedding-3-small",
    keep_original = FALSE
  ) %>%
  step_normalize(all_numeric_predictors())  # Normalize embedding dimensions

# Define model
log_reg_spec <- logistic_reg() %>%
  set_engine("glm") %>%
  set_mode("classification")

# Create workflow
sentiment_workflow <- workflow() %>%
  add_recipe(embedding_recipe) %>%
  add_model(log_reg_spec)

# Fit the model
fitted_workflow <- fit(sentiment_workflow, data = train_data)

# Make predictions on test data
predictions <- predict(fitted_workflow, test_data) %>%
  bind_cols(test_data)

# Evaluate
predictions %>%
  metrics(truth = sentiment, estimate = .pred_class)

Advanced Options

Controlling Embedding Dimensions

Some models support dimension reduction for faster processing:

recipe_spec <- recipe(sentiment ~ text, data = reviews) %>%
  step_foundry_embed(
    text,
    model = "text-embedding-3-small",
    dimensions = 256,  # Reduce from 1536 to 256
    keep_original = FALSE
  )

Lower dimensions mean: - Faster model training - Less memory usage - Some loss in semantic precision

Multiple Text Columns

Process multiple text columns independently:

# Data with multiple text fields
data <- tibble(
  title = c("Great Product", "Terrible Experience"),
  description = c("Works as expected", "Broke immediately"),
  outcome = c(1, 0)
)

recipe_spec <- recipe(outcome ~ ., data = data) %>%
  step_foundry_embed(title, model = "text-embedding-3-small",
                     prefix = "title_") %>%
  step_foundry_embed(description, model = "text-embedding-3-small",
                     prefix = "desc_") %>%
  step_rm(title, description)  # Remove original text columns

Keeping Original Columns

Sometimes you want both the text and embeddings:

recipe_spec <- recipe(sentiment ~ text, data = reviews) %>%
  step_foundry_embed(
    text,
    model = "text-embedding-3-small",
    keep_original = TRUE  # Keep the text column
  )

# Useful when you also want to apply other text processing

Custom Column Prefix

Control the naming of embedding columns:

recipe_spec <- recipe(sentiment ~ text, data = reviews) %>%
  step_foundry_embed(
    text,
    model = "text-embedding-3-small",
    prefix = "embed_"  # Columns will be embed_001, embed_002, etc.
  )

Resampling, caching, and cost

step_foundry_embed() calls the embedding API when a recipe is prepared and when new data is baked. In a resampling workflow, each fold prepares its own recipe. That means the assessment and analysis sets can be embedded repeatedly across folds unless you cache or precompute embeddings.

The default, cache = "none", does not read or write a disk cache. To reuse embeddings for the same text, model, and dimensions, opt into cache = "disk". Without an explicit cache_dir, the cache stays inside tempdir() for the current R session. For a separate temporary workflow, use cache_dir <- tempfile("foundryR-cache-"); after the workflow finishes, remove it with unlink(cache_dir, recursive = TRUE). For persistent reuse, choose a directory you intend to keep. foundry_cache_clear(cache_dir) removes cached embedding files from that directory.

For large or repeated experiments, another option is to embed the text once with foundry_embed() or foundry_embed_batch(), keep the resulting numeric columns, and resample those embeddings:

embedded_reviews <- foundry_embed(
  reviews$text,
  model = "text-embedding-3-small"
)

embedding_matrix <- do.call(rbind, embedded_reviews$embedding)
embedding_cols <- as_tibble(embedding_matrix, .name_repair = "unique")

precomputed <- bind_cols(
  reviews["sentiment"],
  embedding_cols
)

precomputed[, 1:4]

Use the recipe step when preprocessing needs to be self-contained. Precompute when cost, rate limits, or repeated resampling runs matter more.

Cross-validation

Embeddings are generated during prep(), so cross-validation follows the usual tidymodels recipe lifecycle:

# Create CV folds
folds <- vfold_cv(train_data, v = 5, strata = sentiment)

# Fit resamples
cv_results <- fit_resamples(
  sentiment_workflow,
  resamples = folds,
  metrics = metric_set(accuracy, roc_auc)
)

# Collect metrics
collect_metrics(cv_results)

Hyperparameter tuning

Tune the embedding dimensions alongside model hyperparameters:

# Recipe with tunable dimensions
tunable_recipe <- recipe(sentiment ~ text, data = train_data) %>%
  step_foundry_embed(
    text,
    model = "text-embedding-3-small",
    dimensions = tune(),  # Will be tuned
    keep_original = FALSE
  ) %>%
  step_normalize(all_numeric_predictors())

# Model with tunable parameters
rf_spec <- rand_forest(
  mtry = tune(),
  trees = 500,
  min_n = tune()
) %>%
  set_engine("ranger") %>%
  set_mode("classification")

# Workflow
tunable_workflow <- workflow() %>%
  add_recipe(tunable_recipe) %>%
  add_model(rf_spec)

# Define grid
grid <- grid_regular(
  dimensions(range = c(128, 512)),  # Embedding dimensions
  mtry(range = c(10, 50)),
  min_n(range = c(2, 10)),
  levels = 3
)

# Tune only after estimating the API calls and cost.
tune_results <- tune_grid(
  tunable_workflow,
  resamples = folds,
  grid = grid,
  metrics = metric_set(accuracy, roc_auc)
)

# Best parameters
show_best(tune_results, metric = "roc_auc")

Performance considerations

API Rate Limits

Embedding generation makes API calls for each text. For large datasets:

1. Use batch processing with foundry_embed_batch() outside the recipe for large training sets.
2. Precompute and store embeddings for frequently used datasets.
3. Reduce cross-validation folds when repeated API calls are not worth the extra precision.

Cost management

Each embedding call incurs API costs. Strategies to manage costs:

• Start with smaller dimension sizes during development
• Use a subset of data for initial experimentation
• Precompute embeddings for production datasets

Memory usage

With 1,536 dimensions per text and thousands of observations, memory can grow quickly:

# Estimate memory for 10,000 texts
n_texts <- 10000
n_dims <- 1536
bytes_per_double <- 8

memory_mb <- (n_texts * n_dims * bytes_per_double) / 1024^2
print(paste(round(memory_mb), "MB for embeddings alone"))

Consider dimension reduction for large datasets.

Troubleshooting

"Column already exists" error

If you run prep() multiple times, column names may conflict:

# Use a unique prefix if reusing recipes
recipe_spec <- recipe(sentiment ~ text, data = reviews) %>%
  step_foundry_embed(text, model = "my-model",
                     prefix = paste0("v", format(Sys.time(), "%H%M%S"), "_"))

Rate limit errors

If you hit rate limits during prep:

# Prepare in smaller batches
small_sample <- reviews %>% slice_sample(n = 100)
prepped <- prep(recipe_spec, training = small_sample)

Missing credentials

Ensure credentials are set before preparing or baking recipes. These configuration and network checks are not run during rendering:

# Check setup
foundry_check_setup()

# Set credentials if needed.
foundry_set_endpoint(Sys.getenv("AZURE_FOUNDRY_ENDPOINT"))
foundry_set_key("your-api-key")

Next steps

• Learn about Text Embeddings in depth
• Explore Content Safety for responsible AI
• Read the tidymodels documentation for more preprocessing options

if (run_api) {
  httptest2::end_vignette()
}

foundryR documentation built on Sept. 25, 2026, 1:10 a.m.