Annotating open-ended survey responses end to end

In foundryR: Data Frame Workflows for 'Microsoft Foundry'

fixture_dir <- "annotation-workflow"
recording <- nzchar(Sys.getenv("FOUNDRY_RECORD_DOCS"))
have_fixtures <- dir.exists(fixture_dir) && length(list.files(fixture_dir)) > 0
run_api <- requireNamespace("httptest2", quietly = TRUE) &&
  (recording || have_fixtures)

# Attach foundryR before start_vignette(): httptest2 only sources the package's
# inst/httptest2/start-vignette.R (which sets replay placeholders) from attached
# packages.
library(foundryR)

if (run_api) {
  httptest2::start_vignette(fixture_dir)
}

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = run_api
)

Open-ended survey responses are valuable because respondents can say what the researcher did not anticipate. They are also expensive to code by hand. This vignette shows a foundryR workflow that keeps each model step visible in a tibble: extract structured labels, run the same prompt at scale with Batch, embed text for similarity work, and check that generated findings are grounded in the source responses.

Example data

library(foundryR)
library(dplyr)

responses <- tibble::tibble(
  respondent_id = 1:6,
  response = c(
    "The lectures were clear, but the weekly quizzes felt rushed.",
    "I liked the examples in R. More office hours would help.",
    "The project made the material practical.",
    "I struggled because the instructions changed late.",
    "The instructor explained regression well.",
    "The course needed more examples before the final exam."
  )
)

Extract structured annotations

Start with a JSON Schema. Keep the schema small enough that a human reviewer can understand it.

annotation_schema <- list(
  type = "object",
  properties = list(
    sentiment = list(type = "string", enum = c("positive", "negative", "mixed")),
    primary_theme = list(
      type = "string",
      enum = c("instruction", "assessment", "support", "materials")
    ),
    needs_followup = list(type = "boolean"),
    short_summary = list(type = "string")
  ),
  required = c(
    "sentiment",
    "primary_theme",
    "needs_followup",
    "short_summary"
  ),
  additionalProperties = FALSE
)

foundry_extract() uses strict JSON Schema mode by default for supported models. The result is one row per response, with schema fields as columns.

annotations <- foundry_extract(
  responses$response,
  schema = annotation_schema,
  instructions = paste(
    "Code each survey response for a course evaluation.",
    "Use the respondent's words. Do not infer facts that are not stated."
  )
)

coded <- bind_cols(responses, annotations)
coded

Move the same job to Azure Batch

Interactive extraction is useful while designing the schema. For larger jobs, write a JSONL request file and submit it to Azure's Batch API. The submission below requires Azure credentials and is not run during rendering. Its temporary request file is removed after upload.

jsonl <- tempfile(fileext = ".jsonl")

foundry_batch_requests(
  responses,
  input = "response",
  path = jsonl,
  model = "gpt-5-nano",
  endpoint = "/v1/responses",
  body = list(
    instructions = paste(
      "Code each survey response using the supplied schema.",
      "Return only JSON that conforms to the schema."
    ),
    text = list(
      format = list(
        type = "json_schema",
        name = "CourseEvaluationAnnotation",
        schema = annotation_schema,
        strict = TRUE
      )
    )
  )
)

file <- foundry_file_upload(jsonl, purpose = "batch")
unlink(jsonl)
batch <- foundry_batch_create(file$file_id, endpoint = "/v1/responses")
foundry_batch_get(batch$batch_id)

The Batch API is the right choice when the schema is stable and the job is large enough that lower cost and asynchronous execution matter more than immediate feedback.

Embed responses for clustering and near-duplicate checks

Embeddings turn text into numeric vectors. For open-ended survey data, use them to find near-duplicate answers, cluster themes that were not in the original codebook, or build semantic search over the responses.

embeddings <- foundry_embed(
  responses$response,
  model = "text-embedding-3-small"
)

similarity <- foundry_similarity(embeddings)
head(similarity, 10)

High-similarity pairs are useful audit targets. They can reveal duplicate responses, repeated complaints, or places where the schema splits similar answers into different labels.

Validate generated findings with groundedness

After coding and embedding, a researcher often writes a summary. Treat that summary as a claim and check it against the source responses.

finding <- paste(
  "Students generally praised clear instruction and practical examples.",
  "Several asked for more examples and more support before assessments."
)

grounding_text <- paste(responses$response, collapse = "\n")

groundedness <- foundry_groundedness(
  text = finding,
  grounding_sources = grounding_text,
  query = "What did students say about the course?",
  task = "QnA"
)

groundedness

If grounded is FALSE, inspect ungrounded_segments before sharing the finding. This does not replace human review, but it gives you an auditable check inside the same R workflow.

Review table

review <- coded |>
  select(
    respondent_id,
    response,
    sentiment,
    primary_theme,
    needs_followup,
    short_summary
  )

review

The rendered table and chart below summarize the same extraction results when the suggested gt and ggplot2 packages are installed.

review |>
  dplyr::mutate(
    needs_followup = ifelse(needs_followup, "Review", "No review"),
    response = substr(response, 1, 70)
  ) |>
  gt::gt() |>
  gt::cols_label(
    respondent_id = "ID",
    response = "Response",
    sentiment = "Sentiment",
    primary_theme = "Theme",
    needs_followup = "Follow-up",
    short_summary = "Summary"
  ) |>
  gt::tab_header(title = "Structured annotations from open-ended responses") |>
  gt::tab_options(table.font.names = "Inter")

theme_counts <- review |>
  dplyr::count(primary_theme, sentiment, name = "responses")

ggplot2::ggplot(
  theme_counts,
  ggplot2::aes(x = primary_theme, y = responses, fill = sentiment)
) +
  ggplot2::geom_col(position = "dodge", width = 0.72) +
  ggplot2::scale_fill_manual(
    values = c(
      negative = "#D13438",
      mixed = "#FFB900",
      positive = "#107C10"
    )
  ) +
  ggplot2::labs(
    title = "Extracted labels make response patterns countable",
    x = "Primary theme",
    y = "Responses",
    fill = "Sentiment"
  ) +
  ggplot2::theme_minimal(base_size = 12) +
  ggplot2::theme(
    legend.position = "bottom",
    panel.grid.minor = ggplot2::element_blank()
  )

The workflow leaves a trail: raw response, extracted labels, model metadata, embedding similarity, and groundedness checks. That trail is the reason foundryR returns tibbles instead of hiding results inside client objects.

if (run_api) {
  httptest2::end_vignette()
}

foundryR documentation built on Sept. 25, 2026, 1:10 a.m.