A fully open, verification-first LLM-inference tile. One transformer
decoder layer in real RTL — attention, KV-cache compression, softmax,
RMSNorm, RoPE, SwiGLU, residual — every block bit-exact against an
executable golden model, running real Qwen models through the verified
pipeline, and brought up on real FPGA hardware. Pre-silicon, and every
number in this repo is labeled as what it is: measured or projected.
TL;DR for the impatient: GPUs are built to compute. This tile is
built to remember. It does the attention math in hardware and
compresses the conversation's memory (the KV cache) in the same datapath,
so reading speed is designed to stay flat as the context grows. Verified
numbers live one click away: STATUS.md ·
results index · master table.
Large-model inference has two costs that dominate everything else at the
edge: moving weights and remembering context. The first is a
bandwidth problem. The second — the KV cache — is the quietly growing one:
every token you've read must be stored and re-read on every new token, so a
long conversation turns an inference engine into a memory-traffic engine.
Most accelerators treat the KV cache as a software problem: quantize it on
the CPU/GPU, store it, hope. APEX's bet is architectural: put the KV
codec inside the datapath, so keys and values are compressed the moment
they are produced and decompressed the moment they are consumed — with an
importance unit that watches which parts of the context actually matter and
spends bits where they count.
The second bet is methodological: nothing ships without a bit-exact
golden model. Every RTL block in this tree is verified against an
executable NumPy reference — not "close", not "within tolerance":
bit-identical, with mutation-tested testbenches and machine-generated
evidence under an anti-fabrication rule. See
§4.
The scope is deliberately honest: this is one tile — no DRAM
controller, no PCIe, no NoC (out of scope by charter). The architecture is
demonstrated end-to-end on real Qwen2.5-0.5B on FPGA hardware. The
architecture is sized for 7B-class models (head_dim = 128 exists in RTL,
and Qwen2.5-7B tokens have run through the software-verified golden
pipeline — not through silicon), and the
paper-architecture that wraps the tile into a full chip is specified with
per-number provenance in docs/spec/APEX7B_SPEC.md.
Built with Sigmantic AI
APEX was developed with the help of Sigmantic AI's
verification tooling. Sigmantic AI builds autonomous verification agents that
take hardware from natural-language spec to verified RTL — real UVM
testbenches, real simulators, honest sign-off. Underneath is the Polaris
Engine: a closed-loop planner that treats verification as scientific
discovery, choosing at every step the one simulation that will reveal the
most about your design.
2. The architecture
One decode step through the tile
A single new token enters as an activation vector x; the tile produces
the layer output y. Everything between is hardware:
Six ideas make this a tile rather than a pile of blocks:
One GEMM engine, many roles. The MXE (INT8 systolic array) is
time-multiplexed across all seven matrix jobs of a decoder layer — the
QKV projections, the two attention products, the output projection, and
the FFN matrices. The sequencer (§ below) choreographs it.
The KV cache lives compressed. K and V never touch SRAM in fp16.
They are quantized in-flight by the KVQ engine (§3)
and dequantized in-flight on the read side of the attention products.
Online softmax. Scores stream through the ASU's numerically-stable
online softmax — no materialized score matrix, no context-length-sized
buffer.
Importance is a first-class signal. The TIP unit tracks which cached
tokens keep mattering and drives the codec's precision tiers per region
(KVQ8 / KVQ4 / KVQ4+outliers) — the "adaptive" in adaptive precision.
Scales travel with data. Quantization scales for activations and
composites move through the same seams as the tensors they describe, so
a job is bit-exact reproducible from its descriptor alone.
Everything is a job. Each hardware operation is a self-describing
descriptor (matrix shapes, routing, quantization contract). That is what
makes host-driven and walker-driven execution the same machine — and
what makes every flight replayable in simulation, bit for bit.
The deep-dive is ARCHITECTURE.md — including the
design-decision register and the module provenance history.
The layer walker: from job server to sequencer
The tile has two execution modes, and the difference is the project's
central bring-up story:
HOST MODE (proven first) WALKED MODE (the destination)
────────────────────────── ───────────────────────────────
host builds each job host kicks ONCE per layer
│ descriptor, one at a time │
▼ ▼
┌────┐ job ┌──────┐ ┌────────────┐ fetches its own
│host│ ─────► │ tile │ │ seq walker │ descriptors, walks
└────┘ ◄───── └──────┘ │ (on-tile) │ QKV→attn→OPROJ→
result, each job └─────┬──────┘ residual→FFN→DOWN
(~ms of host latency ┌─────▼──────┐
per job, ~100× tax) │ engines │ chained on-tile via
└────────────┘ the XBR fabric
In host mode every job pays a host round-trip — correct, measurable, and
roughly two orders of magnitude of throughput left on the table. The
layer walker (rtl/seq, design notes in
docs/design/B1_WALKER.md) replaces the host
inside a layer: it fetches descriptors, sequences the engines, routes
intermediate tensors through the fabric, and grades composite scales at
every seam — so one kick executes a whole layer.
Walker status is tracked honestly in
docs/design/MASTER_TABLE.md and
docs/design/PROMPT_ON_CHIP.md: the
walked layer is bit-exact in simulation (attention, norm, residual,
FFN and DOWN composition), host-mode is proven on FPGA silicon, and
walked attention on the FPGA is in active bring-up — a genuinely rare
synthesis-vs-simulation defect was root-caused there (the story is worth
reading: docs/results/prompt_on_chip/).
Feeding the beast: weight streaming
A real model's weights don't fit on a tile — they stream. The IB-FUEL lane
(docs/design/IB_FUEL.md) gives the walker a
fuel line from DDR: per-job weight records prefetched into an on-tile FIFO,
so the GEMM engine contracts over streamed weights without stalling, and
per-chunk records let the FFN's big matrices interleave with compute. The
W4 lane (docs/design/W4_DATAPATH.md)
halves that traffic again: weights travel as 4-bit groups and are unpacked
to INT8 at the feeder, with group scales riding the same transport as the
data they describe.
3. The KV-compression engine
The codec is the architectural heart. Per-channel INT4 keys, per-token
INT4 values, an fp16 outlier lane for the channels that refuse to
quantize, and tiered precision (KVQ8 / KVQ4 / KVQ4+) driven by measured
token importance — the published on-device KV-quantization family (cf.
KIVI, KVQuant), implemented as verified hardware.
What makes this implementation worth reading:
Every overhead is counted. The compression ratio is asserted in the
golden model including tags, outlier lanes, padding, and the scale
bank — the number that survives is the number the SRAM actually sees.
Whole-KV ratios and the method-family ceiling analysis:
STATUS.md.
It's not a sidecar. Compression happens between RoPE and the cache
write; decompression happens inside the attention read path. There is no
fp16 copy anywhere.
Accuracy is measured through the verified codec — the same golden
model the RTL is bit-exact against scores real models on the full
HellaSwag validation set, 0.5B/1.5B/7B, with paired per-document
statistics: docs/results/s4_head2head/ ·
docs/results/s5_eval7b/.
It's on real FPGA hardware at the full shipping configuration
(Lattice ECP5-85F, open toolchain — bitstream committed in-repo):
docs/results/s10_fmax/.
4. How we know it's right
The methodology is the product as much as the RTL is:
spec ──► golden model (NumPy, executable) ──► frozen vectors
│ │
│ bit-exact arbiter ▼
└────────────► RTL ◄──── testbenches + SVA + coverage
│ │
│ └── mutation gates: break the
│ RTL on purpose — the TB
▼ must catch every mutant
composition levels
L1 block → L2 pipeline → L3 full layer vs golden
│
▼
same jobs replayed on FPGA silicon, bit for bit
Bit-exact, not approximately right. Millions of checked values, zero
tolerated mismatches, across every block and every composition level —
current counts are machine-generated into STATUS.md on
every full-suite run (the only source of published counts, by rule).
Mutation-tested testbenches. A green suite only counts if it turns
red when the RTL is deliberately broken — mutants that survive are
treated as verification bugs.
An anti-fabrication rule. Published numbers are machine-extracted
from suite logs; the limitations register
(TRACEABILITY.md) records every known gap, and
claims link to the test and log line that back them.
Sim-to-silicon differential discipline. Every FPGA flight is a
register-op program that also runs in the Verilator twin; hardware
captures are compared bit-for-bit against simulation. This is how a
synthesis-level hardware defect was cornered and root-caused entirely
from differential evidence — the toolchain, not the RTL, was lying
(docs/results/prompt_on_chip/).
5. Real models through the pipeline
Not a toy trace: run_tinynpu.py --prompt streams greedy Qwen2.5-7B
(through the software-verified golden pipeline — the FPGA-measured model
is Qwen2.5-0.5B) —
tokens through the golden fixed-point pipeline — the same executable
arbiter the RTL is verified against — producing coherent text, with
sampled hardware-shaped jobs traced and replayed bit-exact, and the
trace's real-model K/V rows replayed through the KVQ RTL in simulation
with zero mismatches. Full artifact run, scope, and counts:
docs/results/s8_7b_token/.
Accuracy through the verified codec (full 10,042-document HellaSwag,
measured, reproducible from a clone): near-lossless at 0.5B/1.5B; at 7B,
KVQ8 shows no detectable effect at full-set power and KVQ4 a small, real,
quantified cost — which is precisely why the outlier tier exists. Numbers,
statistics, and honest caveats:
docs/results/s5_eval7b/.
6. On real hardware
Two independent hardware existence proofs, different parts, different
toolchains:
Lattice ECP5-85F (open flow): the KV-compression engine placed and
routed at the full shipping configuration; bitstream and P&R report
committed. Routed Fmax disclosed with its full context — including the
retirement of an earlier reduced-parameter figure:
docs/results/s10_fmax/.
AWS F2 (Vivado, VU47P): the complete tile builds clean and flies on
real cloud FPGA silicon. First light — the image loading and answering
its verified CSRs — is documented in
docs/results/f2_firstlight/;
since then the campaign has progressed through host-mode attention
(bit-exact on silicon), walked norm/residual chains, DDR weight
streaming, and the walked-attention bring-up with its forensic defect
hunt — the running log is
docs/design/PROMPT_ON_CHIP.md and
docs/results/prompt_on_chip/.
Run it yourself
The verified reference image is agfi-030a812cd224b409d (A2 recipe,
15.625 MHz tile; registered with its build receipt in
scripts/fpga/f2/clock_key.py). These steps
were executed verbatim against that image before this section was committed —
battery 193 checks, 0 fails, then live prompts answered with every walked
value graded bit-exact:
Measured throughput, harness-printed on silicon: 0.56 tok/s on the
fastest registered image (A0, 62.5 MHz — steady 1.78 s/token, confirmed on
two builds, all gates passing) and 0.25 tok/s on this reference image
(A2, 15.625 MHz) — the full per-optimization ladder from the 0.004 tok/s
host-driven baseline (a 140× measured climb) lives in
docs/results/prompt_on_chip/FIRST_WALKED_TOKENS.md.
# prove the silicon: boots an f2.6xlarge, loads the image, flies the
# 193-check battery + walked chains, prints the verdict, terminates itself
bash scripts/fpga/f2/run_walked_demo.sh agfi-030a812cd224b409d
# talk to it: an interactive prompt CLI on the walked pipeline
# ("The capital of France is" -> " Paris.", graded end to end)
bash scripts/fpga/f2/run_chat_demo.sh
Requirements: AWS CLI credentials with F2 access (f2.6xlarge,
us-west-2) and the team's model-weight artifacts (the DDR images derive
from mlx-community/Qwen2.5-0.5B-Instruct-4bit; the staging bucket is
configured in the scripts — external users should regenerate the weight
image with `scripts/fpga/f2/make_weight_image.py
APEX 是一个开放的、以验证为先的 LLM 推理硬件设计：用真正的 RTL 编写的一个 transformer 解码器层，可在 FPGA 上运行 Qwen2.5-0.5B，每个电路模块都经过与软件黄金模型逐位一致的校验。