# hyperflare-inference (NVIDIA/hyperflare-inference)

> Flare is a local inference server built specifically for NVIDIA SM120/SM121 GPUs, designed for long conversations, coding agents, and repeated tool use. It is a research prototype released as open source to gather early feedback and enable collaboration.

- GitHub: https://github.com/NVIDIA/hyperflare-inference
- GitHub.Locker page: https://github.locker/NVIDIA/hyperflare-inference
- Stars: 817
- Language: Python
- License: Apache-2.0
- Last push: 2026-11-18

## Questions and answers

Written by GitHub.Locker from the README and the repository; check the README below for details.

**Is this a full replacement for something like vLLM?**

No. vLLM covers a much broader range of models, hardware and deployment patterns; Flare deliberately trades that breadth for a focused serving contract around selected models and local, single-machine workloads.

**How do I get it running?**

Start with model preparation and serving, using the CLI guide for configuration; the core design doc explains the architectural reasoning if you want more depth.

**Do I need specific GPU hardware?**

Yes. It targets NVIDIA SM120/SM121 GPUs found in workstations and compact systems such as DGX Spark; it's not a general multi-vendor inference server.

**Is this ready for production use?**

No, the project explicitly calls itself a research prototype made available to collect early feedback and enable technical collaboration, not a production-grade serving system.

**Can a coding agent just read this README to use Flare?**

Not directly - the README tells coding agents to skip it and read AGENTS.md instead for setup and usage instructions.

**Why not just use vLLM for local serving?**

Flare is aimed at making the most of a single workstation's memory, bandwidth, and compute for long, repeated-tool-use sessions, treating conversation state (including recurrent state and KV caches) as something that can be reused, branched, and resumed - a narrower goal than vLLM's broad production coverage.

**What are Ignition and Helios?**

Ignition is the Python/PyTorch build-time stage that compiles models and captures CUDA graphs into a runtime package; Helios is the small Rust runtime that serves that package, handling scheduling, cache storage, and request handling without Python.


---

## README (checked against GitHub 2026-11-18)

Source: https://github.com/NVIDIA/hyperflare-inference/blob/main/README.md

<p align="center">
  <img src="docs/images/hyperflare.png" alt="Flare — ASCII wordmark above a supermassive solar flare" width="912">
</p>

> **Coding agents:** skip this README and read [AGENTS.md](AGENTS.md) directly.

Hyperflare is a local inference server for NVIDIA SM120/SM121 GPUs, built for
long conversations, coding agents, and repeated tool use. Its goal is to make
the memory, bandwidth, and compute in workstations and compact systems such
as DGX Spark go further, while keeping generation responsive.

This is a research prototype, and is made available via open-source to enable
technical collaboration and collect early feedback.

The design rests on a few principles:

- **Embrace the standard PyTorch ecosystem at build time; serve from Rust.**
  Ignition uses PyTorch, Transformers, and FlashInfer/b12x to load models,
  tune kernels, compile forwards, and capture CUDA graphs into a runtime core.
  Helios, the Rust runtime, serves that core without Python. Compilation and
  graph capture finish before serving.

- **Build on the model ecosystem.** Reuse canonical model implementations
  wherever their dataflow fits, and specialize the parts that need different
  layouts or execution. Model development keeps the tools and libraries that
  make the Python ecosystem productive.

- **Make memory and state explicit.** Plan buffers and execution capacities
  ahead of time. Treat conversation state—including recurrent state alongside
  KV caches—as a whole, so reuse, branching, and resuming preserve it correctly.

- **Keep the runtime small and shared.** Model recipes own model computation;
  the runtime owns scheduling, cache storage, and request handling. A focused
  hardware and kernel stack keeps those boundaries manageable.

[vLLM](https://docs.vllm.ai/en/latest/) covers a much broader range of models,
hardware, and deployment patterns. Hyperflare trades that breadth for a focused
serving contract around selected models and local workloads. Hyperflare is not, and
will never be, a replacement for production inference engines like vLLM.

Think of Hyperflare as a specialized option when its hardware and model coverage
fit your machine.

Start with [model preparation and serving](docs/model-preparation.md).
For native Windows development and serving prebuilt model cores, follow
[Windows serving](docs/windows.md#build-and-test-on-windows).
See the [CLI guide](docs/cli-design.md) for configuration and the
[core design](docs/core-design.md) for the architectural reasoning.

## Licensing and contributions

Hyperflare's project code is licensed under [Apache 2.0](LICENSE). Contributions
require the sign-off described in [CONTRIBUTING.md](CONTRIBUTING.md).
Third-party attributions are listed in [NOTICE](NOTICE); upstream terms remain
applicable to third-party code and assets.

The PyTorch-derived image resizing implementation in `helios/` retains the
[PyTorch BSD-style license](helios/licenses/PyTorch.txt), also distributed with
[Heliopause](heliopause/licenses/PyTorch.txt). The DeepSeek recipe assets retain
their [MIT license](ignition/recipes/deepseek41/assets/LICENSE). The
`heliopause/LICENSE` and `heliopause/NOTICE` symlinks refer to the root files for
Python package distribution.

Upstream chat templates are fetched with the model checkpoints and annotated
locally using bundled insertion patches. Gemma uses pinned Google sources.
The custom Kimi segmented renderer is included as project code. Downloaded
model assets and exported model cores retain their upstream terms; see
[frontend preparation](docs/model-development.md#sampling-and-frontend-metadata).

