needle-2-esp32 README

The project's README from github.com/andrisgauracs/needle-2-esp32. As one markdown file: README.md.

Needle 2 on an ESP32-S3

A 45M-parameter language model doing grammar-guaranteed tool calling entirely on a microcontroller. No network, no cloud. The weights are memory-mapped from flash; you type a request in a terminal and the board's LED does what you asked.

Needle 2 running on an ESP32-S3

Two requests, back to back. Each one streams its reasoning, emits a schema-valid tool call, and drives the onboard RGB LED. Nothing here touches a network. The clip is sped up — each response takes ~40 s on the board.

This is an independent C99 implementation of the .cact inference format — about 3,700 lines with no dependencies. It is not derived from Cactus's C++ engine, which has no ESP32 build.

Hardware

This targets one specific board. No attempt is made to be portable across ESP32 variants; the point was to make one board work well.
Chip ESP32-S3 (Xtensa LX7, dual core)
Flash 16 MB — the model alone needs 13.1 MB
PSRAM 8 MB octal (an "N16R8"-class module)
LED WS2812 addressable RGB on GPIO48
Console over the UART bridge USB port

A 2 MB or quad-PSRAM part will not work — the KV cache alone is 3.5 MB. If your board differs, see Troubleshooting before assuming the code is broken.

Performance

Measured on the board above, at 240 MHz:
Per token 534 ms (1.87 tok/s)
Request, reasoning off ~25 s
Request, reasoning on ~47 s
One-time boot prime ~51 s

The workload is compute-bound, not memory-bound: a GEMV reading weights from PSRAM takes the same time as from flash, and runtime scales linearly with CPU clock. Faster flash/PSRAM settings buy nothing.

Setup

Needs ESP-IDF v5.5+ and Python 3.

# 1. host tools and the reference implementation
python3 -m venv .venv && .venv/bin/pip install numpy rich pyserial \
    huggingface_hub sentencepiece

# 2. fetch the 13.7 MB model (not committed)
.venv/bin/python -c "
from huggingface_hub import hf_hub_download; import shutil, os
os.makedirs('model', exist_ok=True)
shutil.copy(hf_hub_download('Cactus-Compute/needle2','needle2.cact'),
            'model/needle2.cact')"

# 3. host engine, for rehearsing without the board
cmake -S host -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j8

# 4. firmware -- ESP-IDF must be on PATH first
. $HOME/esp/esp-idf/export.sh
cd esp32/needle_demo
idf.py set-target esp32s3 && idf.py build
idf.py -p /dev/cu.usbmodemXXXX flash

# 5. the model, into its own flash partition (~3 min)
cd ../..
esptool.py --chip esp32s3 -p /dev/cu.usbmodemXXXX -b 921600 \
    write_flash 0x210000 model/needle2.cact

Find your port with ls /dev/cu.usb* (macOS) or ls /dev/ttyUSB* (Linux). The model only needs reflashing if you change it -- firmware iterations are just step 4.

Then:

.venv/bin/python tools/needle_tui.py --serial /dev/cu.usbmodemXXXX

The board primes its prompt prefix once (progress bar with an ETA), pulses the LED green three times, and waits for input. --local runs the same UI against the host engine if you want to try it without hardware.

Using your own tools

Nothing in engine/ needs to change — the grammar is compiled from your schema at runtime. Three steps:

1. Describe the tool in tools/led.json. This is the single source of truth; the firmware's C string is generated from it at build time.

[{"name":"set_servo","description":"Rotate a servo",
  "parameters":{"type":"object","properties":{
    "channel":{"type":"integer","minimum":0,"maximum":3},
    "degrees":{"type":"integer","minimum":0,"maximum":180}},
  "required":["channel","degrees"]}}]

2. Write a handler in esp32/needle_demo/main/main.c, between the YOUR TOOLS START HERE / END HERE markers:

static void tool_set_servo(const char *args)
{
    int ch  = (int)arg_num(args, "channel", 0);
    int deg = (int)arg_num(args, "degrees", 90);

    printf("ACT servo channel=%d degrees=%d\n", ch, deg);
    fflush(stdout);
    servo_write(ch, deg);            /* your hardware */
}

arg_str, arg_num and arg_bool read fields. The call is grammar-constrained, so fields are guaranteed present, correctly typed, and within any declared minimum/maximum — no validation needed.

3. Register it in the same file:

static const struct { const char *name; tool_fn fn; } TOOL_TABLE[] = {
    { "set_servo", tool_set_servo },
};

Rebuild and flash. Print one ACT <what> k=v ... line so the TUI can show what happened, or ACT none if nothing matched.

What the model can and cannot do

It is a dispatcher, not a chatbot — every reply is a tool call or the empty call []. No world knowledge, no free text.

Constraints worth knowing before designing a schema:

Troubleshooting

Three settings account for nearly every problem on this hardware. All three fail silently, so check them first.

No serial output, but the board is clearly running. The console is on the UART bridge, not USB-Serial/JTAG. Leave CONFIG_ESP_CONSOLE_* at its default. Setting CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG=y produces a board that runs and emits nothing, while ROM boot messages still appear over the same port — which makes it look like a crash.

Boot loop with no error message. PSRAM line mode. 8 MB parts are octal:

CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_SPEED_80M=y

Quad mode gives E quad_psram: PSRAM chip is not connected, or wrong PSRAM line mode — but only once the console works, which is why the two failures together are hard to unpick.

It runs but is ~1.5x too slow. ESP-IDF defaults the ESP32-S3 to 160 MHz. CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y is the documented maximum, not an overclock.

idf.py monitor needs a TTY; use tools/mon.py in scripts.

Want a timing breakdown? Uncomment the ND_PROFILE line in esp32/needle_demo/components/needle/CMakeLists.txt for per-phase milliseconds at boot. It is off by default because the timers cost a little speed.

Verifying a build

The engine is checked against independent references, so a fork can prove it is still correct:

.venv/bin/python tools/check_engine.py model/needle2.cact ./build/nd_dump
.venv/bin/python tools/check_tokenizer.py model/needle2.cact ./build/nd_dump
./build/nd_gtest
cd tools && ../.venv/bin/python ref_forward.py ../model/needle2.cact --int8-kv
Check Expected
CQ2/CQ4 dequant + GEMV vs NumPy < 3e-07
Tokenizer vs SentencePiece exact, 15/15
27-layer forward vs NumPy reference ~2.5e-05, identical argmax
Grammar 18/18

ref_forward.py is a NumPy re-implementation of the architecture reading the same .cact blob — full-sequence and non-incremental, so comparing against it also validates the KV cache and its ring buffer.

Layout

engine/     portable C99 engine, no dependencies
  nd_cact.c       .cact v3 container reader
  nd_quant.c      Cactus-Quants kernels (pair-LUT GEMV, FWHT)
  nd_tokenizer.c  SentencePiece BPE
  nd_model.c      Simple Attention Network forward pass
  nd_grammar.c    JSON schema -> byte-level grammar
  nd_sample.c     grammar-constrained sampling
esp32/needle_demo/  ESP-IDF app: mmap, WS2812, serial protocol
host/               CLI harness and grammar tests
tools/              TUI, validation scripts, canonical tool schema

Credits

Needle 2 and its weights are by Cactus Compute, Apache 2.0. This engine is an independent implementation of the published .cact format and the architecture described in arXiv:2607.18363.

needle-2-esp32 (andrisgauracs/needle-2-esp32)

Searching Database…

198 ·GitHub
Needle-2-esp32
What is this?It is a C firmware that runs Cactus's Needle 2, a 45-million-parameter language model, entirely offline on an ESP32-S3 microcontroller, turning plain-English requests into guaranteed-valid JSON tool calls.
Is this a portable ESP32 library or built for one exact board?It targets one specific board configuration only (ESP32-S3 with 16MB flash and 8MB octal PSRAM) and makes no attempt to support other ESP32 variants; it is also an independent ~3,700-line C99 reimplementation of the .cact inference format, not derived from Cactus's existing C++ engine.
How do I get it running?Install ESP-IDF v5.5+ and Python 3, find your serial port (ls /dev/cu.usb on macOS or ls /dev/ttyUSB on Linux), then build and flash the firmware; the board primes its prompt once and pulses the LED green three times before it accepts input over the console.
What exact hardware does this need?An ESP32-S3 (dual-core Xtensa LX7) with 16MB flash and an 8MB octal PSRAM module ('N16R8'-class); a board with 2MB flash or quad-mode PSRAM will not work because the model alone needs 13.1MB and the KV cache needs 3.5MB.
What can and can't the model actually do?It is a pure dispatcher, not a chatbot: every reply is either a tool call or an empty call, with no free text or world knowledge, and its 256-token context window realistically fits only 2-3 tool schemas before accuracy degrades.
Does any data leave the board?No. The model weights are memory-mapped from flash and inference runs entirely on-device with no network connection at all.
How do I add a custom tool for the board to call?Describe the tool's schema in tools/led.json (the single source of truth), write a handler in main.c between marked comment blocks using argstr/argnum/argbool to read fields, register it, then rebuild and flash.
Why does the wording of a tool's description matter so much?Descriptions directly shape the model's behavior — the README notes that the phrase 'solid or blink' on a mode field correctly triggers 'blink', while a more verbose rewording of the same field caused the model to get it wrong.
How is the custom inference engine checked for correctness?It's validated against independent references: dequantization and GEMV are checked against NumPy (under 3e-07 error), the tokenizer matches SentencePiece exactly, a full forward pass matches a NumPy reimplementation, and grammar constraints pass 18/18 tests.
What's the most common reason the board seems broken but isn't?Three silent-failure settings account for most issues: using the wrong console UART setting, running PSRAM in quad instead of octal mode, and leaving the CPU at its default 160MHz instead of the documented 240MHz maximum.
What are similar projects?cardputer-ai, esp32-ai, ESP32s3-LLM-Cluster, needle, mimiclaw, coyopedal, muse-gadget-sdk, openTPU
Which topics is it in?esp32-s3, function-calling, on-device-ai, microcontroller, esp32, tool-calling, edge-ai, embedded-systems, firmware, iot, llm-inference, embedded
Creator
andrisgauracs
Size
0.7 MB
Stars
198
Updated
CommitAug 18
Age
2mo
Commits · 90 Days