are: An R Eval

In vitals: Large Language Model Evaluation

An R Eval

Description

An R Eval is a dataset of challenging R coding problems. Each input is a question about R code which could be solved on first-read only by experts and, with a chance to read documentation and run some code, by fluent data scientists. Solutions are in target() and enable a fluent data scientist to evaluate whether the solution deserves full, partial, or no credit.

Pass this dataset to Task$new() to situate it inside of an evaluation task.

Usage

are

Format

A tibble with 29 rows and 7 columns:

id

Character. Unique identifier/title for the code problem.

input

Character. The question to be answered.

target

Character. The solution, often with a description of notable features of a correct solution.

domain

Character. The technical domain (e.g., Data Analysis, Programming, or Authoring).

task

Character. Type of task (e.g., Debugging, New feature, or Translation.)

source

Character. URL or source of the problem. NAs indicate that the problem was written originally for this eval.

knowledge

List. Required knowledge/concepts for solving the problem.

Source

Posit Community, GitHub issues, R4DS solutions, etc. For row-level references, see source.

Examples

are

dplyr::glimpse(are)

vitals documentation built on June 24, 2025, 9:08 a.m.