NVIDIA has a new public repository, flare-local-inference, created on 20 October under Apache-2.0. Its README describes Flare as “a local inference server for NVIDIA SM120/SM121 GPUs, built for long conversations, coding agents, and repeated tool use,” aimed at workstations and compact systems “such as DGX Spark.” We found no NVIDIA announcement; the repository is the source.
Python to prepare, Rust to serve
The design splits the work in two. A preparation tool called Ignition uses PyTorch, Transformers and FlashInfer/b12x to load a model, tune kernels, compile forwards and capture CUDA graphs into a runtime package. Helios, which the README calls “the Rust runtime, serves that package without Python.” All compilation and graph capture finish before serving starts.
The model preparation guide adds detail:
- A runtime package is a signed bundle of content-addressed blobs: weights, graphs, kernels and frontend assets. The final format uses a signed envelope with encrypted cubins and graph records, and unsigned staging packages cannot load in Helios.
- Every recipe defaults to FP8 E4M3 KV-cache storage. The guide names Qwen, GLM and DeepSeek recipes, each keeping some model-specific formats.
- Each tensor-parallel size needs its own package. The guide says Qwen at TP1 and TP2 and GLM at TP4 have been run end to end, and Qwen at TP4 still needs validation.
- Conversation state, including recurrent state alongside KV caches, is treated as one unit so reuse, branching and resuming keep it intact. That is the part aimed at agents that come back to the same long context.
What it does not claim
The README is unusually plain about limits. “vLLM covers a much broader range of models, hardware, and deployment patterns. Flare trades that breadth for a focused serving contract around selected models and local workloads.” It adds that Rust gives the runtime explicit resource ownership, but “performance still depends on the GPU work, memory traffic, and scheduling,” and tells readers to “compare the engines on your actual model and workload.”
Two cautions. The hardware scope is SM120 and SM121 only. And GitHub lists Python as the repository’s main language: the Rust is the serving runtime, not the whole project. For another take on local agent inference, see Magnitude.
Update, 12 November 2026: Flare is now Hyperflare, with a first v0.1.0 release
NVIDIA’s repository is now NVIDIA/hyperflare-inference, and the project calls itself Hyperflare; the old flare-local-inference address redirects there. We found no announcement of the rename, so we only report that the project is now published under that name.
Its first tagged release, v0.1.0, was published at 19:31 UTC on 11 November. The notes list more than 80 merged pull requests and ten first-time contributors (GitHub logins; we did not check who employs them). Highlights from the notes:
- Prefix caching across machines. Disk prefix caching across hosts, a prefix KV cache shared between instances and hosts through a pool manager, and the host cache tiers moved into a separate crate,
selene, that does not need CUDA. - Multi-host serving. Preparing and serving over hosts cabled as a chain, and prefix cache pools for multi-host tensor parallelism with opt-in RoCE on x86_64.
- More model recipes. A Gemma 4 text recipe with multi-token prediction, DeepSeek V4.1 work including numeric reasoning effort and an opt-in MXFP8 draft head, NVFP4 draft heads for Qwen3.8 tensor parallelism, and Kimi K3.1 changes.
- Windows. Preparing, exporting and serving packages on Windows on RTX Spark N1X.
- Usage reporting. Reasoning token counts in Responses and Chat usage, and a hard reasoning-token budget.
The README framing has not softened. It calls Hyperflare “a research prototype” and says it “is not, and will never be, a replacement for production inference engines like vLLM.”

The Campfire
CommentsThanks. Your comment is waiting for the desk. Once it is approved it appears here within about a minute.
Add a comment