NEWS.md

In localLLM: Running Local LLMs with 'llama.cpp' Backend

localLLM 1.4.1

localLLM 1.4.0 was released on GitHub only. The CRAN update from 1.3.0 is 1.4.1, which includes all of the 1.4.0 changes below. After updating the package, run install_localLLM() to download the 1.4.1 backend.

Bug Fixes

• download_model(show_progress = FALSE) no longer prints download progress. The backend ignored the argument and always printed progress, for model_load() with a URL too. With show_progress = TRUE, progress is now printed once per 5% step. It used to repeat each step many times, which wrote megabytes to Rscript logs and knitr output when a download sat at 0% before the first 5%.

localLLM 1.4.0

Backend

• Upgraded the llama.cpp backend from b8766 to b11869 (April → October 2026, about 3,100 upstream commits). It adds 38 new model architectures, including DeepSeek V3.2 and V4, GLM-5.3-Flash (GLM5-Next), Kimi-K3, MiniMax-M3, MiniMax-01, Qwen3.8-Flash-Next, Cohere2 MoE, BailingMoE 3, Granite SWA, Hunyuan-VL and Gemma Embedding 2, and it adds the Q2_0 quantization type. After updating the package, run install_localLLM() to download the new backend.

• Faster parallel generation — generate_parallel() is about 35% faster with the new backend. Sequential generation speed is unchanged. Measured with 100 classification prompts on an Apple M3 Pro (Metal), n_seq_max = 10: Gemma 4 E2B went from 43.9 s to 28.0 s, and Qwen3 4B from 68.3 s to 44.8 s.

• Faster CPU inference on macOS — CPU weight repacking is now enabled in the macOS builds. It had been disabled because the package build compiled its own copies of the CPU backend without llama.cpp's CPU flags. CPU-only generation (n_gpu_layers = 0) with Q4_0 models is about 30% faster on Apple Silicon: Gemma 4 E2B went from 39 to 51 tokens/s on an M3 Pro. Metal (GPU) results are unchanged.

• macOS 13.3 or later is now enough — the macOS backend libraries are built for macOS 13.3+, as llama.cpp's own releases are. The 1.3.0 libraries required macOS 14 (Apple Silicon) or macOS 15 (Intel), because they inherited the OS version of the build machine.

Bug Fixes

• Chat-template markers in prompts are now tokenized as special tokens (#6, #7, contributed by t-davidson). Markers such as <|im_start|> used to be split into ordinary text pieces, so every templated prompt that generate(), generate_parallel() and quick_llama() passed to the model was malformed. Small models were hit hardest: SmolLM2-135M returned empty strings for many prompts, and Qwen2.5-0.5B leaked |im_end| fragments into its answers.

• generate_parallel() no longer fails on recurrent and hybrid models (Qwen3.5, Qwen3-Next, LFM2, Granite 4 hybrid, Nemotron-H, Mamba, RWKV, ...). Prompts returned "[ERROR] Failed to decode prompt tokens" whenever the shared prefix of a batch covered a whole prompt. That happened in four cases:

• a single prompt;
• identical prompts;
• the last prompt when length(prompts) %% (n_seq_max - 1) == 1;
• every prompt when n_seq_max <= 2.

Prefix sharing now follows the rule llama.cpp's own server uses: every prompt keeps at least one token of its own to decode. Outputs of other models are unchanged.

• model_metadata() is now exported. It was documented and announced in 1.3.0, but was missing from NAMESPACE, so it could only be called as localLLM:::model_metadata().

• model_load() accepts paths that start with ~, and a cache_dir under ~. Such paths used to fail with "Cannot open model file".

• No backend log lines at verbosity = 0 when contexts and models are freed. After model_load() or context_create(), logging fell back to llama.cpp's default, which prints every level. Freeing a context later (rm() + gc(), or quitting R) then printed ~llama_context: ... compute buffer size ... and ggml_metal_free: deallocating. Logging between calls now follows the most recently used verbosity, as it already did after generate().

Behavior Changes

• Outputs can differ from 1.3.0 for the same prompt and seed. The main reason is that prompts are now tokenized correctly (see Bug Fixes). On macOS, CPU-only outputs can also differ for some quantization types (for example Q4_K_M and Q8_0), because CPU repacking is now enabled. Record the package and backend version alongside your results; see the Reproducible Output vignette.

• Special-token text inside your data is now parsed as special tokens. If a document you pass to the model contains a string such as <|im_end|>, that string is now treated as the model's control token, as in llama.cpp's own server, and it can end or reshape the response. Remove or escape such strings when you process raw text.

• A negative repeat_last_n is now an error in generate(), generate_parallel() and quick_llama(). llama.cpp no longer supports -1 ("whole context"); the new backend would silently disable the repetition penalty instead. quick_llama() checks the value before it loads or downloads a model.

• use_mmap and use_mlock in model_load() must be TRUE or FALSE. NA used to reach the backend as TRUE. With the default use_mmap = TRUE, the backend now uses llama.cpp's automatic load mode, which memory-maps the model but falls back to a normal read on devices that do not support mmap.

Known Issues

• R-level informational messages from model_load(), such as "Using cached model: ...", are still shown at verbosity = 0; only verbosity < 0 suppresses them.

Internal

• NAMESPACE is now generated by roxygen2. The hand-maintained file is how model_metadata() went unexported.
• A new CI smoke test (scripts/smoke_test.R) runs a real model against all 7 platform builds. It checks tokenization, generate(), and generate_parallel(), including the prefix-sharing edge cases.
• Fixed the CI static-library symbol check, which had never passed.

localLLM 1.3.0

New Functions

• model_metadata(model) — returns all GGUF key-value metadata as a named character vector. Useful for inspecting model architecture, quantization type, and embedded chat template. Example: model_metadata(model)["tokenizer.chat_template"].

Bug Fixes

• Fixed apply_chat_template() failing for Gemma 4 models — Gemma 4 uses a <|turn> / <turn|> chat format not recognized by llama_chat_apply_template() (returns -1). The fallback now calls common_chat_templates_apply() from common/chat.cpp, which executes the Jinja2 template embedded in the model's GGUF directly. This works for any model with a valid Jinja2 template regardless of the C API whitelist. enable_thinking defaults to true, so Gemma 4 generates thinking content naturally without a pre-closed thought block. Tool calls and multimodal content are not handled.

• Fixed stop token leaking in generate() and generate_parallel() for ChatML models (OLMo, Llama 3) — Two separate issues fixed:

• Windowed find: OLMo tokenizes <|im_end|> as 6 separate pieces, with the last piece being >\n (merging > and newline). The previous exact-suffix check failed because the response ended with <|im_end|>\n instead of exactly <|im_end|>. Changed to a windowed find() that searches within the last stop.size() + 4 bytes and truncates at the match position. Applied to both generate() and generate_parallel().
• <|start_header_id|> loop: Llama 3.2 3B sometimes omits <|eot_id|> and jumps directly to <|start_header_id|> to begin a new turn, causing infinite repetition. Added <|start_header_id|> to text_stop_strings in both functions.
• generate()'s stop list also expanded from {"<turn|>", "<end_of_turn>"} to {"<turn|>", "<end_of_turn>", "<|eot_id|>", "<|im_end|>", "<|start_header_id|>"} to match generate_parallel().

Backend

• Upgraded llama.cpp backend from b8664 → b8766 — 102 builds of improvements. Gemma 4 audio conformer encoder support, various bug fixes. The Gemma 4 template issue is not fixed upstream in this version; our local fallback is the active workaround.

Bug Fixes

• Fixed verbosity not forwarded in quick_llama() — verbosity parameter was accepted but silently dropped when passed through to .generate_single() and .generate_multiple(), so backend logging level had no effect during quick_llama() calls. Now correctly forwarded to generate() and generate_parallel().

• Fixed backend errors crashing R instead of being catchable — All Rcpp::stop() calls in src/interface.cpp replaced with Rf_error(). stop() throws a C++ exception which crosses the C boundary (.Call() registration) and triggers std::terminate(), killing the R process. Rf_error() uses longjmp which R's condition system can intercept, so tryCatch() now works correctly for all backend errors including the OOM guard.

• Fixed model-loading progress dots leaking to stderr with verbosity = 0 — llama_model_load_from_file() has its own progress_callback that prints dots to stderr independently of the log callback system. Now set to a no-op when verbosity < 2 in localllm_model_load_safe(). Model loading is fully silent at the default generation verbosity.

Behavior Changes

• generate_parallel(progress) now defaults to interactive() — previously defaulted to TRUE, which printed carriage-return-based progress bars to log files and R CMD check output. The new default shows the progress bar only in interactive R sessions and suppresses it in scripts and automated checks.
• quick_llama(progress) now defaults to interactive() — same rationale as above; no effect on single-prompt calls.

API Changes

• Removed quick_llama(stream) parameter — the stream argument was present in the function signature but was never passed to any downstream function (it was placeholder code with a comment "available for future use"). Removed to avoid user confusion.

Backend

• New localllm_set_verbosity() C API — added to the backend binary and wired through the proxy layer (proxy.h/cpp, interface.cpp, init.cpp). Enables per-call verbosity control at the C level (integer 0–3, negative = fully silent). Called automatically by generate(), generate_parallel(), model_load(), and context_create() before each C invocation.

• C-layer OOM crash guard in localllm_model_load_safe() — added a last-resort memory check that fires even when check_memory = FALSE. If the model file is larger than total physical RAM, the function now returns a clean error ("Model file (X.X GB) exceeds total physical RAM (Y.Y GB)...") instead of proceeding to llama_model_load_from_file() and letting macOS OOM-kill the R process silently. The guard only blocks provably-impossible loads (file size > total RAM) and does not interfere with the existing R-layer check. Supported on macOS (sysctl hw.memsize), Linux (/proc/meminfo MemTotal), and Windows (GlobalMemoryStatusEx).

Known Issues

• R-layer model_load() messages not suppressed by verbosity = 0 — Two R-level message() calls in api.R ("Using cached model: ..." and the GPU/unified-memory info line) print unconditionally regardless of verbosity. The verbosity parameter controls only the C backend log level; these R-layer informational messages are a separate code path not yet gated on verbosity. Confirmed against Gemma 4 26B-A4B (IQ2_XXS) on 2026-04-12.

Documentation

• Verbosity dual-default design now documented — generate() and generate_parallel() roxygen entries now explain why they default to 0L (called in loops, per-call logs would be noisy) and cross-reference model_load()/context_create() (default 1L, run once per session, warnings should be visible).

localLLM 1.3.0

Performance Fix: Parallel Generation Speedup Restored

• Fixed generate_parallel() performance regression introduced by llama.cpp b7825's new memory API
• The llama_memory_seq_cp() call was dropped during the b7825 migration, causing every parallel slot to re-decode the full prompt instead of sharing the prefix
• Restored prefix sharing via full-range copy (p0=-1, p1=-1), which is compatible with the new API
• Benchmark result: parallel generation is now ~1.4–1.6x faster than sequential (was 0.85x — slower — before this fix)

Platform Support

• Added Intel Mac binary — generate() and generate_parallel() now work on Intel Macs (x86_64); GPU acceleration is not available on Intel Mac, CPU inference is used
• Fixed hardware_profile() crash on Linux and Windows when GPU diagnostic tools (nvidia-smi, rocm-smi, clinfo) are not installed

Backend Upgrade: llama.cpp b7825 -> b8664

• Upgraded llama.cpp backend from b7825 (Jan 2026) to b8664 (Apr 2026)
• ~840 builds of improvements, bug fixes, and new model support

New Model Support (18 new architectures)

• Gemma 4, Qwen 3.5, Qwen 3.5 MoE, ERNIE 4.5
• Granite (standalone), Granite MoE, Granite Hybrid
• JAIS-2, DeepSeek OCR, GLM-DSA, EuroBERT
• Kanana-2, PaddleOCR-VL, ARWKV7, PLM
• BailingMoE, DOTS1, Arcee, AFMoE

New Chat Templates

• DeepSeek OCR template
• Granite 4.0 template (with tool call support)

Build System

• Added vendor/cpp-httplib dependency (required by updated common/ library)
• Added license embedding support via cmake/license.cmake

API Compatibility

No changes to R-level API - All existing R code continues to work without modification.

localLLM 1.2.1

Bug Fixes

• Fix CRAN NOTE: redirect model cache to tempdir() during automated checks so that R CMD check no longer creates ~/.cache/R/localLLM in the home directory (CRAN policy violation).
• Update hardware_profile() example to use \donttest{} instead of if (interactive()) guard, per CRAN best practices.

localLLM 1.2.0

Major Changes

Backend Upgrade: llama.cpp b5421 → b7825

• Upgraded llama.cpp backend from b5421 (Dec 2024) to b7825 (Jan 2025)
• ~400 commits of improvements and architectural changes
• Latest stable release as of January 2025

Core Architecture Migration: KV Cache → Unified Memory API

Breaking changes in backend (transparent to R users): - Migrated from llama_kv_self_* API to llama_memory_* API - Supports heterogeneous model architectures: - Standard Transformers (LLaMA, Qwen, Mistral, etc.) - Mamba/RWKV (State Space Models) - Hybrid models (Jamba, LFM2) - Sliding Window Attention (Qwen2-MLA)

Key improvements: - Better memory management and automatic defragmentation - Enhanced support for parallel inference with shared prefixes - Improved reproducibility of generation results - More efficient batch processing

Batch API Modernization

• Updated to new batch construction API:
• Old: llama_batch_get_one()
• New: llama_batch_init() + common_batch_add() + llama_batch_free()
• Better memory safety and resource cleanup

Improvements

Memory Management

• Automatic memory cleanup on errors to prevent state corruption
• Enhanced reproducibility: Each generate() call starts from clean state
• Thread configuration: Added n_threads_batch parameter for batch processing

Error Handling

• Improved error recovery with automatic memory state cleanup
• Better error messages for decode failures

Performance

• Optimized batch processing with independent thread configuration
• Automatic defragmentation (no manual intervention needed)
• Better Metal GPU utilization on macOS

API Compatibility

No changes to R-level API - All existing R code continues to work without modification:

library(localLLM)

backend_init()
model <- model_load("model.gguf")
ctx <- context_create(model, n_ctx = 512)
result <- generate(ctx, "Hello", max_tokens = 10)
# All existing code works exactly the same

Backend Library Changes

Compilation

• New build script: backend/llama.cpp/build_localllm.sh
• Improved dependency management (OpenSSL, Metal, Accelerate)
• Better symbol visibility and library linking

File Modifications

Updated files: - custom_files/localllm_capi.cpp (10 locations modified) - Memory API migration (8 locations) - Batch API modernization (2 locations) - Error handling improvements - Thread configuration updates

Unchanged: - custom_files/localllm_capi.h (C API interface) - All R layer code (R/*.R) - Proxy layer (src/proxy.cpp) - Test suite (tests/testthat/*.R) - Documentation

Testing

• ✅ All 206 tests pass
• ✅ R CMD check passes with no errors
• ✅ Compiled library size: 4.2 MB
• ✅ 35 exported C API functions verified

Installation Notes

First-time Installation

install.packages("localLLM_1.2.0.tar.gz", repos = NULL, type = "source")
library(localLLM)
install_localLLM()  # Will download the new b7825 backend

Upgrading from 1.1.0

remove.packages("localLLM")
install.packages("localLLM_1.2.0.tar.gz", repos = NULL, type = "source")
library(localLLM)
install_localLLM(force = TRUE)  # Force reinstall backend

Documentation

New technical documentation: - UPGRADE_COMPLETE.md - Complete upgrade report - CRITICAL_CHANGES_REQUIRED.md - Detailed change checklist - MIGRATION_ANALYSIS_b5421_to_b7785.md - Full migration analysis - Architecture deep-dive in planning documents

Known Issues

• CRAN check warning about unchanged version number (resolved in this release)
• Metal shader embedding required for GitHub Actions (already implemented)

Future Enhancements

Potential optimizations for future releases: - Flash Attention support for improved performance - Unified Buffer optimization for multi-sequence inference - SWA (Sliding Window Attention) for ultra-long contexts (128K+)

Contributors

• Backend migration and testing: Claude (Anthropic)
• Package maintenance: Yaosheng Xu
• Original package: Eddie Yang

For more information about llama.cpp, see: - llama.cpp releases - llama.cpp documentation

localLLM 1.1.0

Previous release notes (if any) would go here...

localLLM documentation built on Nov. 3, 2026, 1:09 a.m.