gemma4.c README

The project's README from github.com/qmog-ai/gemma4.c. As one markdown file: README.md.

gemma4.c

Gemma 4 E2B CPU inference in 700 lines of pure C.

An educational project made to understand how LLM inference works. The full inference path is implemented in one file without external libraries.

For a detailed walkthrough of the implementation, see the blog post.

Quick start

gemma4.c runs on Linux and Windows. You need a CPU with AVX2, an OpenMP-capable C compiler, and make. The model takes about 5.0 GB of disk space, and 8 GB of RAM is recommended.

Clone the repository and download the ready-to-run model:

git clone https://github.com/qmog-ai/gemma4.c
cd gemma4.c
python3 -m pip install -U huggingface_hub
hf download QmogAI/gemma4-e2b-int8 gemma4-E2B-int8.bin --local-dir .

On Linux:

make
./run -t 0 -n 256 "Why is the sky blue?"

On Windows, use a MinGW-w64 environment that provides gcc, OpenMP, and make:

make win64 WINCC=gcc
.\run.exe -t 0 -n 256 "Why is the sky blue?"

Options

Model

gemma4.c runs google/gemma-4-E2B-it-qat-q4_0-unquantized, the unquantized QAT checkpoint Google released for Gemma 4 E2B. The C runtime cannot read that checkpoint directly. exporter.py takes its tokenizer and language-model weights and writes them in the exact layout used by gemma4.c.

Matrix weights are stored as int8 with FP16 scales. Inputs to linear layers are dynamically quantized to int8 while the rest of the activations remain float32. The resulting file is about 5.0 GB (4.7 GiB).

To create it yourself instead:

python3 -m pip install -r requirements.txt
python3 exporter.py /path/to/gemma-4-E2B-it-qat-q4_0-unquantized -o ./gemma4-E2B-int8.bin

Python is only needed to export the model or run numerical validation. Once the .bin file exists, inference runs entirely through the C program.

Benchmark

Measured on an AMD Ryzen 7 7700 using the default native build.

Prefill (tok/s)

Prompt tokens gemma4.c (int8) llama.cpp (Q8_0)
512 630.73 272.69
2,048 494.14 258.56
8,192 254.02 223.99
16,384 106.36 188.60

Decode (tok/s)

Starting context gemma4.c (int8) llama.cpp (Q8_0)
512 25.16 22.67
2,048 24.04 21.28
8,192 20.35 17.82
16,384 16.80 14.26

Prefill measures the time to process the stated number of prompt tokens. Decode first fills the KV cache to the stated depth, then measures 128 single-token steps. Both include the logits needed to produce the next token, but exclude sampling and terminal output. Results are the mean of three timed runs after one discarded warmup. Both implementations use FP32 KV caches, 512-token batches, eight CPU threads, and native builds.

The benchmark command takes the prefill length followed by the number of decode steps:

./run -m ./gemma4-E2B-int8.bin --bench 512 128

Numerical validation

The implementation is validated against the Hugging Face Transformers reference implementation running Google's unquantized Gemma 4 E2B QAT checkpoint in BF16. Both implementations process the complete Éva Gauthier article from the WikiText-103 validation split under teacher forcing, comparing logits at all 2,388 token positions.

Metric Result
Top-1 agreement 2,305 / 2,388 (96.5%)
Mean KL divergence 0.005207

An exact match is not expected because gemma4.c uses int8 matrix weights and linear inputs while the reference runs the unquantized checkpoint in BF16. The output distributions nevertheless remain closely aligned.

Build the C runtime first (make or make win64). The complete validation peaks at about 10 GiB of RAM:

python3 validation.py

Pass a smaller token count for a quicker check:

python3 validation.py 64

validation.py uses the runtime's --dump-logits flag to collect float32 logits after each prompt position. The flag can also be used directly when comparing gemma4.c with another implementation:

./run -m ./gemma4-E2B-int8.bin --dump-logits "Why is the sky blue?" > logits.bin

Repository contents

gemma4.c (qmog-ai/gemma4.c)

cpu
Searching Database…

176 ·GitHub
Gemma4.c
What is this?gemma4.c is a CPU inference runtime for Google's Gemma 4 E2B model written in about 700 lines of pure C, built to show how LLM inference actually works under the hood.
Is this a library I link into my code, or a standalone program?It is a standalone command-line program: you build an executable with make and run it directly with a model path and prompt options.
How do I get it running?Clone the repo, run make (or make win64 on Windows), download the ready-to-run exported model file (about 5.0 GB), and run the resulting binary pointing at that model with -m.
What do I need installed to build and run it?You need a CPU with AVX2 support, an OpenMP-capable C compiler, and make; Python is only needed if you want to export the model yourself or run numerical validation.
Does it work on any CPU, and how much disk/RAM do I need?It requires AVX2 and runs only on Linux and Windows (no macOS/ARM mentioned); the model takes about 5.0 GB of disk space and 8 GB of RAM is recommended, with validation peaking near 10 GiB.
Which operating systems does it support?Linux and Windows (the latter via a MinGW-w64 toolchain with a small compatibility layer for memory mapping).
How fast is it compared to llama.cpp?On an AMD Ryzen 7 7700, gemma4.c's int8 build beats llama.cpp's Q8_0 build on most prefill lengths and decode speeds, e.g. 25.16 vs 22.67 tok/s decode at a 512-token context.
How accurate is the int8 inference compared to the real model?Against Hugging Face Transformers running the unquantized BF16 checkpoint on a WikiText-103 passage, gemma4.c matches the top-1 token 96.5% of the time with a mean KL divergence of 0.005207.
How do I get the model weights into the format this runtime needs?exporter.py converts Google's released unquantized QAT checkpoint (google/gemma-4-E2B-it-qat-q40-unquantized) into the int8-with-FP16-scales binary layout that the C runtime reads directly.
What's inside the repository besides the inference code?gemma4.c holds the tokenizer, model, kernels, and generation loop; exporter.py converts checkpoints; validation.py and validation.txt check numerical accuracy against Transformers; win.c/win.h add Windows memory-mapping support; and a Makefile builds both platforms.
What are similar projects?Strata, jeeves, v100-skinny, turbo-fieldfare, llama.cpp, openTPU, deltafin, StartLux-Decision
Which topics is it in?gemma, model, cpu, llama-cpp, inference-engine, llm-inference, quantization, openai-compatible, c, inference, benchmark, command-line-tool
Creator
qmog-ai
Size
<0.1 MB
Stars
176
Updated
CommitSep 10
Age
1mo
Commits · 90 Days