esp32-thinks README

项目的 README,来自 github.com/raullenchai/esp32-thinks。合并为一个 markdown 文件:README.md。

esp32-thinks

esp32-thinks

MIT License ESP-IDF v6.0.x board v0.2.12 CI

A multi-model Muse — a personal AI assistant on ESP32, talking out of the box.

Think of it as a Muse that thinks with your models: say "Hi, ESP" (or press the knob), ask anything, and it answers out loud — streaming sentence-by-sentence while the model is still writing, in English, Chinese or Korean, on a small gadget with a mic, a speaker and a round display.

No vendor lock on the brain: speech, transcription and chat are three independent endpoints, each pointed at any OpenAI-compatible server — one provider for all three, or a different one per task. Run it bare against QuickSilver, or put the in-repo openmuse agent in front to give it location, weather, time and memory.

Why

Cloud assistants rent you a brain: one vendor's models, one vendor's pricing, one vendor's uptime, and your voice on their servers. The gadget hardware is fine — the lock is in the firmware.

esp32-thinks flips that. The device runs your stack: pick the cheapest transcription, the smartest chat model, the voice you like — per task, changeable in one config line and a rebuild. The implementation is a reviewable patch against a pinned open-source SDK, so you can read it, change it, or point it at a local server. The optional openmuse agent keeps the extra context (location, memory, tools) on your machine too.

What you get

Quick start

# 0) One-time: ESP-IDF v6.0.x for esp32s3 (docs/REFERENCE.md §Prerequisites)

git clone https://github.com/facebookincubator/muse-gadget-sdk.git
git -C muse-gadget-sdk checkout 812c46f
git clone https://github.com/raullenchai/esp32-thinks.git
cd esp32-thinks

# 1) Back up the factory NVS once — your restore path if anything goes sideways
./scripts/backup-nvs.sh ../muse-gadget-sdk /dev/ttyACM1   # use your port: ACM0 or ACM1

# 2) Apply the patch + configure your brain (pick a preset)
./apply.sh ../muse-gadget-sdk --preset quicksilver   # or openai / openmuse / custom

# 3) Build + flash (CH342-safe pacing; allow several minutes)
./build-and-flash.sh ../muse-gadget-sdk /dev/ttyACM1

With a preset + headless Wi-Fi in step 2, that's the whole setup: unplug, re-plug, and talk — press the knob, or just say "Hi, ESP".

Command Does
./scripts/backup-nvs.sh <sdk> [port] Dump the factory NVS partition (0x9000, 0x32000) — do this once
./apply.sh <sdk> [--preset ...] Copy pinned CAs, apply patches/muse-brain.patch, generate provider_config.h (asks for keys, Wi-Fi, wake word)
./build-and-flash.sh <sdk> [port] idf.py build + paced flash (firmware + wake-word model); never interrupt it
python3 scripts/tts_probe.py <sdk> [port] ["question"] Typed chat/TTS probe; opens a serial reader and resets the device
./openmuse/run.sh [--funnel] Optional agent: Docker or Node, --funnel exposes HTTPS via Tailscale

Own brain: openmuse (optional)

QuickSilver alone is the fastest path. If you want the gadget to know your city and time, answer weather questions, and remember things across turns, run the agent that ships in this repo:

./openmuse/run.sh              # first run writes openmuse/.env, then exits
$EDITOR openmuse/.env          # set UPSTREAM_API_KEY + OPENMUSE_TOKEN
./openmuse/run.sh --funnel     # Docker or Node; --funnel exposes HTTPS via Tailscale
./apply.sh ../muse-gadget-sdk --preset openmuse
#   chat URL: https://<host>/v1
#   token:    OPENMUSE_TOKEN

openmuse sits between the device and any OpenAI-compatible LLM: it owns the system prompt, adds location/clock/weather/memory tools, proxies STT (one TLS session for STT + chat), and streams the reply through untouched. TTS still goes directly to QuickSilver. It needs no domain — a Tailscale Funnel gives it a real Let's Encrypt certificate. Memory lives in openmuse/data/memories.json. Details: openmuse/README.md. For the protocol, the CopilotKit/OpenMuse boundary, and troubleshooting, see docs/OPENMUSE.md.

Performance (measured on device)

QuickSilver preset, typed warm turns: 2.2–2.5 s for English/Korean and about 6.8 s for Chinese (longer text, slower synthesis). The nightly soak on 2026-10-19 measured 2.2 s / 6.8 s / 2.8 s (EN/CN/KO). Voice turns add recording/VAD: roughly 3–5 s from end of speech to first sound.

With the openmuse agent in front, add one hop and tool work: typed warm turns measured ~3.9 s, a voice turn ~4.7 s; the first turn after boot can be slower while DNS/TLS warm up. These are individual samples, not latency guarantees — provider load and reply length dominate.

See REFERENCE for historical measurements and configuration, and docs/SOAK.md for the nightly stability check.

How a turn flows

 "Hi, ESP" / knob ──▶ record (pre-roll holds the last second)
                        │  ends: voice VAD (~1.4 s silence) or knob release
                        ▼
              trim to speech ──▶ POST WAV/ADPCM ─▶ STT endpoint (or openmuse proxy)
                        │                         (¼-size upload, keep-alive)
                        ▼
              POST /chat/completions (SSE) ──────────────▶ chat endpoint (or openmuse)
                        │  split at sentence boundaries (ASCII + 。!?…)
                        ▼
              per sentence: POST /audio/speech ─▶ MP3 appends to the buffer
                        │                          already decoding + playing
                        ▼
                 first word out loud while the model is still writing

Connection preparation starts after Wi-Fi comes up. A first question can still pay connection costs if preparation has not finished, or when a longer reply needs a separate speech connection. See REFERENCE for historical measurements and connection behavior.

At a glance

Device Seeed SenseCAP Watcher W1-B (ESP32-S3, 1.45" 412×412 round LCD, ES8311 speaker codec, ES7243E mic, PTT knob) — more boards the SDK supports are candidates
Base firmware muse-gadget-sdk @ 812c46f (2026-10-09) + patches/muse-brain.patch (see what's theirs, what's ours)
ESP-IDF v6.0.x (CI pins espressif/idf:v6.0.3)
Brain Any OpenAI-compatible servers: STT (/audio/transcriptions), chat (/chat/completions, SSE streaming), speech (/audio/speech)
Presets openai (all three) · quicksilver (qwen3.7-flash + whisper-large-v3-turbo + chatterbox/qwen3-tts) · openmuse (in-repo agent for STT + chat, QuickSilver TTS) · custom
Wake word esp-sr WakeNet10 wn10_hiesp ("Hi, ESP", on-chip, model flash partition) — custom phrases trainable via Espressif's TTS Pipeline V3
Build host Linux or macOS with ESP-IDF v6.0.x

Serial console

The USB serial port is a full console, handy for bench testing. Ports can renumber on replug — use the one that answers >status (/dev/ttyACM0 or /dev/ttyACM1):

Input Does
hold knob / d…u simulate PTT down/up
k audible self-test: plays the wake phrase through the speaker, then a full voice turn
>chat=hello? typed turn — same chat/TTS pipeline
>tasks dump every task backtrace (works even while wedged)
>status heartbeats: Wi-Fi, heap, DMA, tunnel, firmware version
z / w sleep / wake
p screenshot over serial

Use >chat= for routine checks and d/u for silent PTT input. Reserve k for a deliberate acoustic wake check; avoid loops that repeatedly play the wake phrase.

Built on muse-gadget-sdk

Credit where it's due — and clarity about the split:

The SDK provides the device framework, LVGL UI and round-LCD drivers, audio stack (codec, resampler, minimp3), input/console plumbing, OTA and partitioning. The patch also wraps peripheral I/O to monitor stalls. This recipe targets a pinned SDK commit so patches apply cleanly.

Ours (patches/muse-brain.patch): the brain session (muse_chat_session.cpp — three-endpoint config, SSE sentence streaming, TTS pipeline, connection warm-up + idle probes, turn cancellation), the wake-word integration (muse_voice.c — direct WakeNet10 on idle chunks, wake→PTT synthesis, VAD end-of-turn, quiet-recording guard, stall coroner), the I2C expander watchdog, the model flash partition and the esp-dl kernel fix, plus the tooling (apply.sh, build-and-flash.sh, NVS backup, probes) and the optional openmuse/ agent.

Nothing else from the Muse cloud is required: the vendor agent works as an optional fallback (its token is optional), and with headless provisioning the device never contacts it.

Configuration

Everything lives in esp32/components/muse/provider_config.h (written by apply.sh):

Setting What
TTS_URL / TTS_MODEL / TTS_VOICE Speech endpoint, model, voice
TTS_MODEL_CJK Optional second speech model, selected for Chinese/Korean reply text
BRAIN_STT_WAV_ADPCM 1 uploads IMA ADPCM WAV at about one quarter of PCM size; enabled for the verified quicksilver and openmuse presets, 0 otherwise
TTS_API_KEY / TTS_CA Its key and pinned root (OPENAI_CA or ISRG_CA)
BRAIN_STT_URL / BRAIN_STT_MODEL Transcription endpoint (shares the chat key); openmuse points this at the agent's /audio/transcriptions proxy
BRAIN_STT_LANG / BRAIN_STT_PROMPT Empty by default: omit both fields and auto-detect language; optionally pin an ISO 639-1 language or provide a prompt
BRAIN_CHAT_URL / BRAIN_CHAT_KEY / BRAIN_MODEL Chat endpoint; leave the URL empty to use the Muse cloud agent. openmuse uses the agent URL and token
BRAIN_CHAT_EXTRA Extra JSON merged into the chat body (provider quirks, e.g. DeepSeek's reasoning.exclude)
BRAIN_SYSTEM Language-following persona; one or two sentences, ~40 words / 60 Chinese characters; greeting-only input returns Please ask. (openmuse replaces this with its own prompt)
WAKE_WORD 1 = "Hi, ESP" wake word + button; 0 = button only
WAKE_VAD_DB End-of-turn silence threshold for wake turns (raise toward -35 if it keeps recording after you stop)
DEV_WIFI_SSID / DEV_WIFI_PASS Headless provisioning; empty = normal BLE setup

Presets: openai, quicksilver, openmuse (STT + chat through the in-repo openmuse agent — location, weather, time, memory — with QuickSilver for TTS), and custom.

Changed a define? Re-run ./build-and-flash.sh — it rebuilds and reflashes.

Security notes

Roadmap

Build details and historical receipts: docs/REFERENCE.md. For contributing and validation, see CONTRIBUTING.md.

Troubleshooting

Top four (full checklist in REFERENCE):

Getting help

Changelog

See CHANGELOG.md and the releases page.

esp32-thinks (raullenchai/esp32-thinks)

正在搜索数据库…

这是什么?这是一个固件补丁,可以把 ESP32-S3 小工具变成个人 AI 语音助手:说出唤醒词或按下旋钮,它就会开口回答,并在模型还在生成内容时逐句流式播报回复。
这是独立应用,还是对现有固件的补丁?这是一个可供审查的补丁(patches/muse-brain.patch),应用在 Meta 开源项目 muse-gadget-sdk 的一个固定提交上,并不是独立应用。apply.sh 会在编译前应用该补丁并生成配置。
怎样才能让它在设备上运行?先用 backup-nvs.sh 备份一次出厂 NVS 分区,再运行 apply.sh --preset 生成密钥、Wi-Fi 和唤醒词配置,然后用 build-and-flash.sh 通过 ESP-IDF 编译,并经 USB 烧录。
需要什么硬件和工具?一块 Seeed SenseCAP Watcher W1-B(带麦克风、扬声器和圆形 LCD 屏幕的 ESP32-S3 设备),一台用于编译的 Linux 或 macOS 电脑(需安装 ESP-IDF v6.0.x),以及至少一家兼容 OpenAI 接口的服务商的 API 密钥。
这能在任何 ESP32 开发板上运行吗?目前只在 Seeed SenseCAP Watcher W1-B 上验证过,其他 muse-gadget-sdk 支持的开发板只是候选。另外请注意,API 密钥会直接编译进固件,因此它适合个人台式小工具,不适合作为共享或公开设备使用。
我的语音数据究竟会发到哪里?你不必把语音交给某一家厂商的云端处理,而是可以按任务分别为语音转文字(STT)、对话和文字转语音(TTS)指定任意兼容 OpenAI 接口的服务器,密钥只保存在你自己编译的固件里。
可选的 openmuse 代理有什么作用?它位于设备和你的大语言模型之间,可以提供位置、时钟、天气和持久化记忆等功能。它还会代理语音转文字,使语音转文字和对话共用同一个 TLS 会话,而文字转语音仍直接连接你的服务商。
为什么没有配套应用,也没有蓝牙配对步骤?Wi-Fi 凭据直接写入配置文件,因此设备首次启动时即可自行完成配网,无需任何应用、蓝牙设置或厂商账号。
支持哪些语言?已测试英语、中文和韩语。语音转文字会自动识别语言,助手用相同语言回复;无法判断时默认使用英语。
语音回复有多快?使用 QuickSilver 预设时,文字输入且已预热的对话轮次约为 2.2-2.5 秒(英语/韩语)和 6.8 秒(中文);实时语音对话每轮会额外增加约 3-5 秒,用于录音和语音活动检测。
大小
0.5 MB
星标
265
时长
1个月
提交 · 90 天