esp32-thinks README

README проекта с github.com/raullenchai/esp32-thinks. Одним файлом markdown: README.md.

esp32-thinks

esp32-thinks

MIT License ESP-IDF v6.0.x board v0.2.12 CI

A multi-model Muse — a personal AI assistant on ESP32, talking out of the box.

Think of it as a Muse that thinks with your models: say "Hi, ESP" (or press the knob), ask anything, and it answers out loud — streaming sentence-by-sentence while the model is still writing, in English, Chinese or Korean, on a small gadget with a mic, a speaker and a round display.

No vendor lock on the brain: speech, transcription and chat are three independent endpoints, each pointed at any OpenAI-compatible server — one provider for all three, or a different one per task. Run it bare against QuickSilver, or put the in-repo openmuse agent in front to give it location, weather, time and memory.

Why

Cloud assistants rent you a brain: one vendor's models, one vendor's pricing, one vendor's uptime, and your voice on their servers. The gadget hardware is fine — the lock is in the firmware.

esp32-thinks flips that. The device runs your stack: pick the cheapest transcription, the smartest chat model, the voice you like — per task, changeable in one config line and a rebuild. The implementation is a reviewable patch against a pinned open-source SDK, so you can read it, change it, or point it at a local server. The optional openmuse agent keeps the extra context (location, memory, tools) on your machine too.

What you get

Quick start

# 0) One-time: ESP-IDF v6.0.x for esp32s3 (docs/REFERENCE.md §Prerequisites)

git clone https://github.com/facebookincubator/muse-gadget-sdk.git
git -C muse-gadget-sdk checkout 812c46f
git clone https://github.com/raullenchai/esp32-thinks.git
cd esp32-thinks

# 1) Back up the factory NVS once — your restore path if anything goes sideways
./scripts/backup-nvs.sh ../muse-gadget-sdk /dev/ttyACM1   # use your port: ACM0 or ACM1

# 2) Apply the patch + configure your brain (pick a preset)
./apply.sh ../muse-gadget-sdk --preset quicksilver   # or openai / openmuse / custom

# 3) Build + flash (CH342-safe pacing; allow several minutes)
./build-and-flash.sh ../muse-gadget-sdk /dev/ttyACM1

With a preset + headless Wi-Fi in step 2, that's the whole setup: unplug, re-plug, and talk — press the knob, or just say "Hi, ESP".

Command Does
./scripts/backup-nvs.sh <sdk> [port] Dump the factory NVS partition (0x9000, 0x32000) — do this once
./apply.sh <sdk> [--preset ...] Copy pinned CAs, apply patches/muse-brain.patch, generate provider_config.h (asks for keys, Wi-Fi, wake word)
./build-and-flash.sh <sdk> [port] idf.py build + paced flash (firmware + wake-word model); never interrupt it
python3 scripts/tts_probe.py <sdk> [port] ["question"] Typed chat/TTS probe; opens a serial reader and resets the device
./openmuse/run.sh [--funnel] Optional agent: Docker or Node, --funnel exposes HTTPS via Tailscale

Own brain: openmuse (optional)

QuickSilver alone is the fastest path. If you want the gadget to know your city and time, answer weather questions, and remember things across turns, run the agent that ships in this repo:

./openmuse/run.sh              # first run writes openmuse/.env, then exits
$EDITOR openmuse/.env          # set UPSTREAM_API_KEY + OPENMUSE_TOKEN
./openmuse/run.sh --funnel     # Docker or Node; --funnel exposes HTTPS via Tailscale
./apply.sh ../muse-gadget-sdk --preset openmuse
#   chat URL: https://<host>/v1
#   token:    OPENMUSE_TOKEN

openmuse sits between the device and any OpenAI-compatible LLM: it owns the system prompt, adds location/clock/weather/memory tools, proxies STT (one TLS session for STT + chat), and streams the reply through untouched. TTS still goes directly to QuickSilver. It needs no domain — a Tailscale Funnel gives it a real Let's Encrypt certificate. Memory lives in openmuse/data/memories.json. Details: openmuse/README.md. For the protocol, the CopilotKit/OpenMuse boundary, and troubleshooting, see docs/OPENMUSE.md.

Performance (measured on device)

QuickSilver preset, typed warm turns: 2.2–2.5 s for English/Korean and about 6.8 s for Chinese (longer text, slower synthesis). The nightly soak on 2026-10-19 measured 2.2 s / 6.8 s / 2.8 s (EN/CN/KO). Voice turns add recording/VAD: roughly 3–5 s from end of speech to first sound.

With the openmuse agent in front, add one hop and tool work: typed warm turns measured ~3.9 s, a voice turn ~4.7 s; the first turn after boot can be slower while DNS/TLS warm up. These are individual samples, not latency guarantees — provider load and reply length dominate.

See REFERENCE for historical measurements and configuration, and docs/SOAK.md for the nightly stability check.

How a turn flows

 "Hi, ESP" / knob ──▶ record (pre-roll holds the last second)
                        │  ends: voice VAD (~1.4 s silence) or knob release
                        ▼
              trim to speech ──▶ POST WAV/ADPCM ─▶ STT endpoint (or openmuse proxy)
                        │                         (¼-size upload, keep-alive)
                        ▼
              POST /chat/completions (SSE) ──────────────▶ chat endpoint (or openmuse)
                        │  split at sentence boundaries (ASCII + 。!?…)
                        ▼
              per sentence: POST /audio/speech ─▶ MP3 appends to the buffer
                        │                          already decoding + playing
                        ▼
                 first word out loud while the model is still writing

Connection preparation starts after Wi-Fi comes up. A first question can still pay connection costs if preparation has not finished, or when a longer reply needs a separate speech connection. See REFERENCE for historical measurements and connection behavior.

At a glance

Device Seeed SenseCAP Watcher W1-B (ESP32-S3, 1.45" 412×412 round LCD, ES8311 speaker codec, ES7243E mic, PTT knob) — more boards the SDK supports are candidates
Base firmware muse-gadget-sdk @ 812c46f (2026-10-09) + patches/muse-brain.patch (see what's theirs, what's ours)
ESP-IDF v6.0.x (CI pins espressif/idf:v6.0.3)
Brain Any OpenAI-compatible servers: STT (/audio/transcriptions), chat (/chat/completions, SSE streaming), speech (/audio/speech)
Presets openai (all three) · quicksilver (qwen3.7-flash + whisper-large-v3-turbo + chatterbox/qwen3-tts) · openmuse (in-repo agent for STT + chat, QuickSilver TTS) · custom
Wake word esp-sr WakeNet10 wn10_hiesp ("Hi, ESP", on-chip, model flash partition) — custom phrases trainable via Espressif's TTS Pipeline V3
Build host Linux or macOS with ESP-IDF v6.0.x

Serial console

The USB serial port is a full console, handy for bench testing. Ports can renumber on replug — use the one that answers >status (/dev/ttyACM0 or /dev/ttyACM1):

Input Does
hold knob / d…u simulate PTT down/up
k audible self-test: plays the wake phrase through the speaker, then a full voice turn
>chat=hello? typed turn — same chat/TTS pipeline
>tasks dump every task backtrace (works even while wedged)
>status heartbeats: Wi-Fi, heap, DMA, tunnel, firmware version
z / w sleep / wake
p screenshot over serial

Use >chat= for routine checks and d/u for silent PTT input. Reserve k for a deliberate acoustic wake check; avoid loops that repeatedly play the wake phrase.

Built on muse-gadget-sdk

Credit where it's due — and clarity about the split:

The SDK provides the device framework, LVGL UI and round-LCD drivers, audio stack (codec, resampler, minimp3), input/console plumbing, OTA and partitioning. The patch also wraps peripheral I/O to monitor stalls. This recipe targets a pinned SDK commit so patches apply cleanly.

Ours (patches/muse-brain.patch): the brain session (muse_chat_session.cpp — three-endpoint config, SSE sentence streaming, TTS pipeline, connection warm-up + idle probes, turn cancellation), the wake-word integration (muse_voice.c — direct WakeNet10 on idle chunks, wake→PTT synthesis, VAD end-of-turn, quiet-recording guard, stall coroner), the I2C expander watchdog, the model flash partition and the esp-dl kernel fix, plus the tooling (apply.sh, build-and-flash.sh, NVS backup, probes) and the optional openmuse/ agent.

Nothing else from the Muse cloud is required: the vendor agent works as an optional fallback (its token is optional), and with headless provisioning the device never contacts it.

Configuration

Everything lives in esp32/components/muse/provider_config.h (written by apply.sh):

Setting What
TTS_URL / TTS_MODEL / TTS_VOICE Speech endpoint, model, voice
TTS_MODEL_CJK Optional second speech model, selected for Chinese/Korean reply text
BRAIN_STT_WAV_ADPCM 1 uploads IMA ADPCM WAV at about one quarter of PCM size; enabled for the verified quicksilver and openmuse presets, 0 otherwise
TTS_API_KEY / TTS_CA Its key and pinned root (OPENAI_CA or ISRG_CA)
BRAIN_STT_URL / BRAIN_STT_MODEL Transcription endpoint (shares the chat key); openmuse points this at the agent's /audio/transcriptions proxy
BRAIN_STT_LANG / BRAIN_STT_PROMPT Empty by default: omit both fields and auto-detect language; optionally pin an ISO 639-1 language or provide a prompt
BRAIN_CHAT_URL / BRAIN_CHAT_KEY / BRAIN_MODEL Chat endpoint; leave the URL empty to use the Muse cloud agent. openmuse uses the agent URL and token
BRAIN_CHAT_EXTRA Extra JSON merged into the chat body (provider quirks, e.g. DeepSeek's reasoning.exclude)
BRAIN_SYSTEM Language-following persona; one or two sentences, ~40 words / 60 Chinese characters; greeting-only input returns Please ask. (openmuse replaces this with its own prompt)
WAKE_WORD 1 = "Hi, ESP" wake word + button; 0 = button only
WAKE_VAD_DB End-of-turn silence threshold for wake turns (raise toward -35 if it keeps recording after you stop)
DEV_WIFI_SSID / DEV_WIFI_PASS Headless provisioning; empty = normal BLE setup

Presets: openai, quicksilver, openmuse (STT + chat through the in-repo openmuse agent — location, weather, time, memory — with QuickSilver for TTS), and custom.

Changed a define? Re-run ./build-and-flash.sh — it rebuilds and reflashes.

Security notes

Roadmap

Build details and historical receipts: docs/REFERENCE.md. For contributing and validation, see CONTRIBUTING.md.

Troubleshooting

Top four (full checklist in REFERENCE):

Getting help

Changelog

See CHANGELOG.md and the releases page.

esp32-thinks (raullenchai/esp32-thinks)

Ищем в базе…

Что это такое?Это патч прошивки, который превращает гаджет на ESP32-S3 в персонального голосового ИИ-помощника: скажите ключевую фразу или нажмите на ручку, и он ответит вслух, проговаривая ответ по предложениям прямо во время того, как модель его генерирует.
Это отдельное приложение или патч к уже существующей прошивке?Это патч, который можно проверить (patches/muse-brain.patch), он применяется к закреплённому коммиту открытого SDK от Meta muse-gadget-sdk, а не является отдельным приложением. Скрипт apply.sh накладывает его и генерирует конфигурацию перед сборкой.
Как запустить это на устройстве?Один раз сохраните заводской раздел NVS командой backup-nvs.sh, затем запустите apply.sh --preset, чтобы сгенерировать конфигурацию ключей, Wi-Fi и ключевой фразы, а потом соберите прошивку через ESP-IDF и загрузите её по USB скриптом build-and-flash.sh.
Какое железо и инструменты мне нужны?Плата Seeed SenseCAP Watcher W1-B (ESP32-S3 с микрофоном, динамиком и круглым LCD-экраном), компьютер на Linux или macOS с ESP-IDF v6.0.x для сборки и API-ключ хотя бы одного провайдера, совместимого с OpenAI.
Это работает на любой плате ESP32?Проверена только плата Seeed SenseCAP Watcher W1-B, остальные платы из muse-gadget-sdk пока лишь кандидаты. Также учтите, что API-ключи вшиваются прямо в прошивку, поэтому это задумано как личный гаджет для экспериментов, а не как общее или публичное устройство.
Куда на самом деле уходят мои голосовые данные?Вместо того чтобы голос обрабатывало облако одного вендора, для каждой задачи (STT, чат и TTS) вы указываете любые совместимые с OpenAI серверы на свой выбор, а ключи хранятся только в вашей собственной сборке.
Что добавляет необязательный агент openmuse?Он стоит между устройством и вашей LLM, добавляя местоположение, время, погоду и постоянную память. Он проксирует STT, поэтому STT и чат используют одну TLS-сессию, а TTS по-прежнему идёт напрямую к вашему провайдеру.
Почему не нужны ни приложение-компаньон, ни сопряжение по BLE?Учётные данные Wi-Fi записываются прямо в файл конфигурации, поэтому устройство настраивается автоматически при первом включении без какого-либо приложения, настройки по BLE или аккаунта производителя.
Какие языки поддерживаются?Протестированы английский, китайский и корейский. Распознавание речи само определяет язык, и ассистент отвечает на том же языке, а если язык неясен, отвечает по умолчанию на английском.
Насколько быстро отвечает голосовой помощник?С пресетом QuickSilver текстовые обмены репликами на уже прогретой системе занимают около 2,2–2,5 с для английского и корейского и 6,8 с для китайского. Голосовые реплики добавляют примерно 3–5 с на запись и определение голосовой активности.
Какие есть похожие проекты?EchoMuse, mimiclaw, cardputer-ai, hermes-gadget-sdk, esp32-ai, vibepulse, jargo, ESP32s3-LLM-Cluster
Автор
raullenchai
Размер
0.5 MB
Звёзды
265
Возраст
1 мес
Коммиты · 90 дней