llama_serve_openai: Serve an OpenAI-compatible HTTP API for a local model

In llamaR: Interface for Large Language Models via 'llama.cpp'

Serve an OpenAI-compatible HTTP API for a local model

Description

Loads a GGUF model once and exposes it over an OpenAI-compatible HTTP API so any OpenAI client (OpenCode, ellmer, the 'openai' Python SDK, …) can talk to it. Implements 'GET /v1/models' and 'POST /v1/chat/completions' (both blocking and 'stream = true'). The HTTP/SSE layer is provided by drogonR; generation runs through llamaR's streaming API (llama_gen_begin / llama_gen_next / llama_gen_end).

Usage

llama_serve_openai(
  model_path,
  port = 11434L,
  n_ctx = 4096L,
  n_gpu_layers = -1L,
  model_id = NULL,
  host = "127.0.0.1",
  template = NULL,
  max_tokens = 512L,
  ...
)

Arguments

model_path	Path to a GGUF model file.
port	Port to listen on. Default 11434 (the Ollama port, so clients pointed at a local Ollama work unchanged).
n_ctx	Context size for the loaded model.
n_gpu_layers	Layers to offload to GPU (-1 = all).
model_id	Identifier reported in /v1/models and echoed in responses. Defaults to the model file's base name.
host	Address to bind. Default "127.0.0.1" (local only).
template	Chat template string, or NULL to use the model's built-in template.
max_tokens	Default max_new_tokens when a request omits it.
...	Reserved for future options.

Details

The server is single-sequence: requests are handled one at a time on the main R thread (each streamed token is one event-loop pump). This is meant for a single local user/agent, not concurrent load.

drogonR is an optional dependency (Suggests); install it with install.packages("drogonR") (or from its repository) before calling this function.

Value

Invisibly NULL. Blocks serving until drogonR::dr_stop() is called (typically from another process or an interrupt).

See Also

[llama_gen_begin], [llama_generate]

Examples

## Not run: 
llama_serve_openai("model.gguf", port = 11434L)
# In another shell, point any OpenAI client at
#   http://127.0.0.1:11434/v1
# e.g. GET /v1/models and POST /v1/chat/completions

## End(Not run)

llamaR documentation built on May 28, 2026, 1:06 a.m.