esp32-ai README

The project's README from github.com/slvDev/esp32-ai. As one markdown file: README.md.

Running a 28.9M parameter LLM on a microcontroller

𝕏 slvDev  ·  LinkedIn

28.9M-parameter LLM running on an ESP32-S3

This is a 28.9 million parameter language model that generates text on an ESP32-S3 microcontroller. It runs on the chip itself, with nothing sent to a server, and it displays generated text at 9.88 tokens per second on a small screen wired to the chip. It fits because most of the model lives in flash instead of RAM, using Per-Layer Embeddings, an idea from Google's Gemma 3n.

The numbers
Parameters 28.9M stored (25M of them in a flash lookup table)
Chip ESP32-S3, 512KB SRAM, 8MB PSRAM and 16MB flash
Speed 9.88 tok/s end to end, 94.9 ms/token of compute
Connectivity none, everything runs on the device
Model size 14.9MB at 4-bit

Why it is hard, and how it fits anyway

A microcontroller has very little fast memory. The ESP32-S3 gives you 512KB of SRAM, and only the values touched many times per token can live there: activations and norm weights. The dense core and output head, scanned once per position, sit in PSRAM. What is left is the embedding tables, and their size is what normally decides how big a model can be.

In this model, most parameters sit in an embedding table, which the model reads from rather than computes on. So that 25-million-parameter table stays in slow flash, and only the few rows each token needs are pulled from it, about 450 bytes. Most of the model is therefore never loaded to run it: it sits in flash and is sampled a little at a time.

That idea is Google's Per-Layer Embeddings, from Gemma 3n. Here it runs on the memory layout of a microcontroller instead of a phone or a GPU.

Each tier holds whatever is read at its own frequency:

  SRAM  (fast, tiny)   activations and norm weights, touched many times a token
  PSRAM (medium)       the core and output head, read once per position
  FLASH (huge, slow)   the 25M-param table, about 6 rows read per token (~450 B)

What it does, and what it does not

The model was trained on TinyStories, so it writes short, simple stories and mostly keeps them coherent. It will not answer questions, follow instructions, write code, or know facts. That limit comes from the small part of the model that does the reasoning, and the memory trick does not change it. What is interesting here is the architecture, fitting a large model onto a tiny chip, rather than what a 28.9 million parameter model can say.

Models

Running it yourself

Download and deployment are separate operations: one reaches the network, the other touches the board.

scripts/fetch_model.sh barista   # download, verify, install into artifacts/
scripts/deploy.sh barista        # generate headers, run gates, compile, flash

tinystories is the other model, and takes the same two commands. Both require the model to be named, because the board holds one at a time and deploying replaces it.

fetch_model.sh checks the inference assets against a SHA-256 and byte size pinned in the script, and cross-checks the release's own metadata.json against those same pins. It installs nothing unless every check passes, so a failed download leaves what you already have untouched. deploy.sh downloads no model: it works from whatever is already in artifacts/<model>/. It does run two of its header tools through uv, which fetches one pinned wheel the first time on a machine that has never cached it.

The firmware details and the boot output to expect live in firmware/esp32_barista/README.md and firmware/esp32_tinystories/README.md. The reusable architecture is in src/; the training, ablation and quantization code that reproduces the published numbers is in research/tinystories/. The full method, the ablations, and the on-chip measurements are written up in RESULTS.md.

Credit

TinyStories is the dataset this trains on: short synthetic stories simple enough that a small model can still learn to write coherently (Ronen Eldan and Yuanzhi Li, Microsoft Research, arXiv:2305.07759). The other half is Per-Layer Embeddings, Google's design from Gemma 3n, which is what lets a big model fit on a small chip.

Andrej Karpathy's llama2.c is the reference for training a small language model and running it in plain C.

Measurements

Detailed measurements and ablations are documented in RESULTS.md.

esp32-ai (slvDev/esp32-ai)

Searching Database…

4.5K ·GitHub
Esp32-ai
这是什么？一个拥有 2890 万参数的语言模型，文本生成完全在 ESP32-S3 微控制器上运行，生成的文字显示在与芯片相连的小屏幕上。
这是软件库还是硬件项目？它是针对一款特定开发板的固件和模型部署项目，而不是通用库：你下载一个预训练模型并刷写到连接了显示屏的 ESP32-S3 上，而不是把代码导入自己的应用。
怎么让模型在开发板上跑起来？运行 fetchmodel.sh 下载经过验证的模型发布包（在安装任何东西之前，它会先核对固定的 SHA-256 和发布元数据），然后运行 deploy.sh 把 artifacts/ 中的内容刷写到 ESP32-S3 上。
需要哪些硬件和工具？一块带 512KB SRAM、8MB PSRAM 和 16MB 闪存的 ESP32-S3 开发板，连接一块小屏幕；另外部署期间需要用“uv”拉取几个固定版本的头文件工具。
这个模型到底能写什么，又不能做什么？它只在 TinyStories 数据集上训练过，能写出简短、大体连贯的故事，但无法回答问题、遵循指令、编写代码或回忆事实。
运行期间会有数据离开设备吗？不会。推理完全在芯片上进行，模型刷写完成后根本不需要任何网络连接。
一个 28.9M 参数的模型为什么能装进只有 512KB RAM 的芯片？它的大部分参数放在一张 2500 万行的嵌入表里，保存在闪存而非 RAM 中，采用了 Google Gemma 3n 的 Per-Layer Embeddings 思路，因此推理时每个 token 只需从闪存读取大约 450 字节。
有哪些模型可以试用？Barista，回答与浓缩咖啡相关的问题；以及 TinyStories，生成简短故事。两者都用同样的两条命令部署。
它生成文本的速度有多快？对于量化为 4-bit 的 14.9MB 模型，端到端速度为每秒 9.88 个 token（每个 token 计算 94.9 毫秒）。
这个项目的灵感来自哪里？它把 Andrej Karpathy 的 llama2.c 小型纯 C 语言模型训练与推理方案，与 Google Gemma 3n 的 Per-Layer Embeddings 相结合，并在 Microsoft 的 TinyStories 数据集上训练。
What are similar projects?cardputer-ai, needle-2-esp32, ESP32s3-LLM-Cluster, hermes-gadget-sdk, jev, mimiclaw, coyopedal, needle
Which topics is it in?karpathy, cpu-inference, esp32-s3, llama2, on-device-ai, microcontroller, esp32, edge-ai, language-model, firmware, iot, llm-inference
Creator
slvDev
Size
1.4 MB
Stars
4.5K
Updated
CommitSep 23
Age
2mo
Commits · 90 Days