gemma4.c README

The project's README from github.com/qmog-ai/gemma4.c. As one markdown file: README.md.

gemma4.c

Gemma 4 E2B CPU inference in 700 lines of pure C.

An educational project made to understand how LLM inference works. The full inference path is implemented in one file without external libraries.

For a detailed walkthrough of the implementation, see the blog post.

Quick start

gemma4.c runs on Linux and Windows. You need a CPU with AVX2, an OpenMP-capable C compiler, and make. The model takes about 5.0 GB of disk space, and 8 GB of RAM is recommended.

Clone the repository and download the ready-to-run model:

git clone https://github.com/qmog-ai/gemma4.c
cd gemma4.c
python3 -m pip install -U huggingface_hub
hf download QmogAI/gemma4-e2b-int8 gemma4-E2B-int8.bin --local-dir .

On Linux:

make
./run -t 0 -n 256 "Why is the sky blue?"

On Windows, use a MinGW-w64 environment that provides gcc, OpenMP, and make:

make win64 WINCC=gcc
.\run.exe -t 0 -n 256 "Why is the sky blue?"

Options

Model

gemma4.c runs google/gemma-4-E2B-it-qat-q4_0-unquantized, the unquantized QAT checkpoint Google released for Gemma 4 E2B. The C runtime cannot read that checkpoint directly. exporter.py takes its tokenizer and language-model weights and writes them in the exact layout used by gemma4.c.

Matrix weights are stored as int8 with FP16 scales. Inputs to linear layers are dynamically quantized to int8 while the rest of the activations remain float32. The resulting file is about 5.0 GB (4.7 GiB).

To create it yourself instead:

python3 -m pip install -r requirements.txt
python3 exporter.py /path/to/gemma-4-E2B-it-qat-q4_0-unquantized -o ./gemma4-E2B-int8.bin

Python is only needed to export the model or run numerical validation. Once the .bin file exists, inference runs entirely through the C program.

Benchmark

Measured on an AMD Ryzen 7 7700 using the default native build.

Prefill (tok/s)

Prompt tokens gemma4.c (int8) llama.cpp (Q8_0)
512 630.73 272.69
2,048 494.14 258.56
8,192 254.02 223.99
16,384 106.36 188.60

Decode (tok/s)

Starting context gemma4.c (int8) llama.cpp (Q8_0)
512 25.16 22.67
2,048 24.04 21.28
8,192 20.35 17.82
16,384 16.80 14.26

Prefill measures the time to process the stated number of prompt tokens. Decode first fills the KV cache to the stated depth, then measures 128 single-token steps. Both include the logits needed to produce the next token, but exclude sampling and terminal output. Results are the mean of three timed runs after one discarded warmup. Both implementations use FP32 KV caches, 512-token batches, eight CPU threads, and native builds.

The benchmark command takes the prefill length followed by the number of decode steps:

./run -m ./gemma4-E2B-int8.bin --bench 512 128

Numerical validation

The implementation is validated against the Hugging Face Transformers reference implementation running Google's unquantized Gemma 4 E2B QAT checkpoint in BF16. Both implementations process the complete Éva Gauthier article from the WikiText-103 validation split under teacher forcing, comparing logits at all 2,388 token positions.

Metric Result
Top-1 agreement 2,305 / 2,388 (96.5%)
Mean KL divergence 0.005207

An exact match is not expected because gemma4.c uses int8 matrix weights and linear inputs while the reference runs the unquantized checkpoint in BF16. The output distributions nevertheless remain closely aligned.

Build the C runtime first (make or make win64). The complete validation peaks at about 10 GiB of RAM:

python3 validation.py

Pass a smaller token count for a quicker check:

python3 validation.py 64

validation.py uses the runtime's --dump-logits flag to collect float32 logits after each prompt position. The flag can also be used directly when comparing gemma4.c with another implementation:

./run -m ./gemma4-E2B-int8.bin --dump-logits "Why is the sky blue?" > logits.bin

Repository contents

gemma4.c (qmog-ai/gemma4.c)

Searching Database…

173 ·GitHub
Gemma4.c
这是什么？gemma4.c 是一个在 CPU 上运行的推理运行时，面向 Google 的 Gemma 4 E2B 模型，仅用约 700 行纯 C 编写，目的是展示大语言模型推理在底层的实际工作原理。
它是我链接到自己代码里的库，还是独立程序？它是一个独立的命令行程序：用 make 构建出可执行文件，然后直接带上模型路径和提示选项运行。
怎么把它跑起来？克隆仓库，运行 make（Windows 上用 make win64），下载开箱即用的导出模型文件（约 5.0 GB），然后用 -m 参数指向该模型来运行生成的二进制文件。
构建和运行需要安装什么？你需要一颗支持 AVX2 的 CPU、一个支持 OpenMP 的 C 编译器和 make；只有当你想自己导出模型或运行数值验证时才需要 Python。
任何 CPU 都能运行吗？需要多少磁盘/内存？它要求 AVX2，且只能在 Linux 和 Windows 上运行（未提及 macOS/ARM）；模型约占 5.0 GB 磁盘空间，建议 8 GB 内存，验证过程峰值接近 10 GiB。
它支持哪些操作系统？Linux 和 Windows（后者通过 MinGW-w64 工具链，并带一个用于内存映射的小型兼容层）。
和 llama.cpp 相比速度如何？在 AMD Ryzen 7 7700 上，gemma4.c 的 int8 版本在大多数预填充长度和解码速度上都胜过 llama.cpp 的 Q8_0 版本，例如在 512 token 上下文下解码速度为 25.16 对 22.67 tok/s。
与原模型相比，int8 推理的准确度如何？以 Hugging Face Transformers 在 WikiText-103 文段上运行未量化的 BF16 检查点作为基准，gemma4.c 有 96.5% 的概率 top-1 token 一致，平均 KL 散度为 0.005207。
怎么把模型权重转换成这个运行时需要的格式？exporter.py 会把 Google 发布的未量化 QAT 检查点（google/gemma-4-E2B-it-qat-q40-unquantized）转换成 int8 加 FP16 缩放因子的二进制布局，C 运行时可直接读取。
除了推理代码，仓库里还有什么？gemma4.c 包含分词器、模型、核心算子和生成循环；exporter.py 负责转换检查点；validation.py 和 validation.txt 用于对照 Transformers 验证数值精度；win.c/win.h 添加 Windows 内存映射支持；另有一个 Makefile 可构建两个平台。
What are similar projects?Strata, jeeves, v100-skinny, turbo-fieldfare, llama.cpp, openTPU, deltafin, StartLux-Decision
Which topics is it in?gemma, model, cpu, llama-cpp, inference-engine, llm-inference, quantization, openai-compatible, c, inference, benchmark, command-line-tool
Creator
qmog-ai
Size
<0.1 MB
Stars
173
Updated
CommitSep 10
Age
1mo
Commits · 90 Days