sophon-realtime README

项目的 README,来自 github.com/adventists-ai/sophon-realtime。合并为一个 markdown 文件:README.md。

Sophon-Realtime v1: hears your tone, speaks with one

English  |  中文

Hears your tone. Speaks with one.

Sophon-Realtime v1 is an open full-duplex voice model family: it listens while you talk, hears how you say things, and answers with a voice that laughs, sighs and breathes when the moment calls for it.

Tag on Hugging Face Flash on Hugging Face Tag on ModelScope Flash on ModelScope Try it online Apache-2.0

Demo

Real conversations with Sophon-Realtime, recorded as they happened: no edits, no speed-up. Turn the sound on.

https://github.com/user-attachments/assets/4d6c74f5-3bbf-4dc2-8b11-f90ff85de4a2

Chinese demo

https://github.com/user-attachments/assets/1d9442ef-84b0-45a3-acd4-2d05a9562397

Highlights

VoiceBench overall, full-duplex models Big Bench Audio

Full-Duplex-Bench v3 tool use, Pass@1 Paralinguistic understanding, public benchmarks combined

NVV-SuperBench, non-verbal sounds in speech, three builds

Self-run unless noted. VoiceBench: official evaluators with a gpt-4o-mini judge; AlpacaEval on the standard 199-question subset. Big Bench Audio: GPT-4.1 judges the transcript of the spoken answer; other models from Artificial Analysis. Full-Duplex-Bench v3: best of several runs; other systems from the FDB v3 paper (Apr 2026). Paralinguistic understanding: same prompts and option rotation for every model. NVV-SuperBench: official prompts and metric, 50 items per class per language, Gemini 3 Flash judge; the three builds speak the same tagged reply texts in the Chelsie voice. Latency and concurrency: one H100 80 GB, users arriving within 10 s.

News

Every release, including fixes: CHANGELOG · GitHub Releases

Three builds, one brain

All three builds share the same brain: one 32B model that listens to your voice, decides when you have finished, and writes the reply. They differ only in how the reply becomes sound.

Tag (recommended) Flash E2E
How it speaks The brain writes the reply with sound tags ([laughter], [sigh], <strong>…</strong> …); an expressive speech model performs them Same tagged reply, spoken by a lighter, faster speech model (one forward pass per 80 ms frame) No speech-token model: a compact talker (about 350M parameters) reads the tagged reply as it is written and generates continuous voice vectors directly; a decoder turns them into sound
Best for Expressive voice: companions, characters, demos Speed and scale: customer service, in-car, many users per GPU Expressive voice with the fastest first audio; our own engine, easy to modify
Non-verbal sounds (NVV-SuperBench, en / zh) 0.80 / 0.86 0.63 / 0.77 0.75 / 0.85
First audio, one user ~0.6 s 0.31 s ~0.25 s
Users per 80 GB GPU (recommended) 16 40 16
Zero-shot voice cloning yes, from a 3–10 s clip (how) yes, from a 3–10 s clip (how) yes, from a 5–12 s clip (how)
GPUs 1 × 80 GB 1 × 80 GB 1 × 80 GB
Serving vLLM-Omni, one command vLLM-Omni, one command own engine or vLLM-Omni, one command
Weights HF · ModelScope HF · ModelScope HF · ModelScope
Code tag/ flash/ own engine: docs/deploy.md · vLLM-Omni: e2e/

Which one? Start with Tag if you want the voice to act. Pick Flash if the number of users per GPU matters most. Pick E2E for the fastest first audio with a voice that still laughs and sighs. Tag and Flash share the same APIs, so you can switch later; E2E runs on its own engine or, since 1.1.4, on vLLM-Omni with the same APIs as Tag and Flash.

4B builds (new in 1.4)

The same three builds with a 4B brain (DuplexJev-4B-Para with a tag-writing adapter), for one consumer GPU. The speech stages and voices are the same as in the 32B builds; there is no image input.

Tag-4B Flash-4B E2E-4B
GPU one 32 GB (uses ~27 GB) one 24 GB (uses ~21 GB) one 32 GB (uses ~31 GB)
Full-Duplex-Bench v1.0 (4-task avg; 32B: 94.0) 94.3 87.7 92.3
First audio, one user (RTX PRO 6000) 0.40 s 0.21 s 0.31 s
Install SIZE=4B bash install.sh bash install.sh + weights SIZE=4B bash install.sh
Weights HF · ModelScope HF · ModelScope HF · ModelScope

The 4B brain keeps the conversation skills (turn-taking, back-channels, barge-in, sound tags at the same rate as the reference data) but knows and reasons much less: VoiceBench 62.4 (32B: 82.8), Big Bench Audio 44.3% (32B: 81.8%). Use it for local demos, companions and characters, and turn-taking research; use 32B or the hosted API when answers must be right. start.sh sees a 4B model and picks its config.

Languages

Talk to it in your own language. Sophon-Realtime listens and answers in the language you speak.

Language Status
Chinese, English Fully supported
Italian, German, Spanish, French, Portuguese Ready for conversation: clear speech, 80–92% of basic questions answered correctly, multi-turn quality close to English
Japanese, Russian Available on the Flash build
Korean and more Coming soon

Internal tests, November 2026: 15 languages × 20 spoken questions, plus multi-turn dialogues from Omnilingua-AudioMC in 8 languages (Portuguese not included). Japanese is a priority language for us, and upcoming releases will keep improving it.

How it works

Quick start

No GPU? Use the hosted API: sign up at api.adventists.ai/console and get 3 hours of free voice chat (counted on Flash). It speaks the OpenAI Realtime protocol at wss://api.adventists.ai/sophon/v1/realtime (HTTP: https://api.adventists.ai/sophon/v1); set model to sophon-realtime-tag, sophon-realtime-flash or sophon-realtime-e2e. A complete Python example is in the API docs.

Quickest (Tag build, one 80 GB GPU):

pip install "sophon-realtime[trt]"   # Python 3.12, Linux, CUDA 13; installs vLLM-Omni, the Tag plugin and TensorRT
sophon-realtime serve           # downloads Sophon-Realtime-32B-Tag (~42 GB) on first run, then serves on port 8000

Keep the [trt]. Without TensorRT the voice runs in plain PyTorch and is about 3× slower under load (16 users: first audio ~2.4 s instead of ~0.75 s, with 1–1.5 s of stutter). sophon-realtime serve now warns at start-up when TensorRT is missing.

Or use the one-command scripts, which set up an isolated environment:

Tag (one 80 GB GPU):

git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh                 # venv + vLLM-Omni + plugin, downloads the weights (~45 GB)
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en

Flash (one 80 GB GPU):

cd sophon-realtime/flash
bash install.sh
modelscope download --model AdventistsAI/Sophon-Realtime-32B-Flash --local_dir ./model   # or: hf download adventists-ai/Sophon-Realtime-32B-Flash --local-dir ./model
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en --voice chelsie

Both serve ws://HOST:PORT/v1/realtime (full duplex, OpenAI-Realtime-style) and POST /v1/chat/completions (audio in, text + audio out). Details: tag/README.md · flash/README.md.

Mac (Apple silicon, 36 GB+ unified memory recommended; runs locally with MLX):

cd sophon-realtime && bash mac/install.sh   # environment + Sophon-Realtime-32B-Flash-4bit-MLX (~24 GB)
~/sophon-mac/sophon --lang en               # talk through your microphone

Details: mac/README.md.

Optional: Live mode (OpenAI GPT-Live compatible, see live/), next to a running engine:

cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh   # GPT-Live clients: ws://HOST:8780/v1/live/sessions

Live mode (OpenAI GPT-Live compatible)

live/ is a gateway that speaks the OpenAI GPT-Live protocol in front of a Tag or Flash engine: GPT-Live clients switch by changing the URL and key. Sophon keeps the voice and hands lookups and actions to your backend through delegation, then speaks the result. Optionally, a larger model can run the back office.

Zero-shot voice cloning (new in v1.1)

Give Sophon a few seconds of someone's voice and it talks to you in that voice, in real time. No fine-tuning, no restart, one GPU. Works in the Tag and Flash builds, and in E2E since 1.3.0.

https://github.com/user-attachments/assets/7f0627b0-fae4-4633-b412-88cb24942ab8

pip install -U "sophon-realtime[trt]"
sophon-realtime download --source modelscope   # or Hugging Face (default)
cp alice.wav model/voices/                      # 3-10 s, one speaker
sophon-realtime serve                           # then set "voice": "alice"

New, replaced and deleted voices take effect on the next request. Barge-in, backchannels and laughs still work in a cloned voice, and a voice cloned from one language can speak another. Everything stays on your server (details). The video also shows Live mode: Sophon hands lookups (weather, news) to your own backend, keeps talking, then says the answer.

Please only clone your own voice or voices you have permission to use. Do not use it to impersonate anyone or to deceive.

Chinese-subtitled version

https://github.com/user-attachments/assets/74bd6096-b6fc-4433-a1d7-d3e967107193

E2E

The E2E build has no speech-token model in the middle. A compact talker (about 350M parameters) reads the brain's tagged reply while it is still being written and generates a stream of continuous voice vectors, about 6.5 frames per second; a 2-step decoder and a vocoder turn them into 24 kHz audio. Laughs, coughs, sighs, breaths and stressed words are produced by the talker itself, as part of the same stream. Everything runs on one 80 GB GPU with our own engine: docs/deploy.md. Weights: HF · ModelScope.

Measured from a fresh install on one H100: first audio for one user in about 0.25 s; 16 users arriving within 10 s get first audio in 0.44 s (median) / 0.56 s (P90) with no reply falling behind playback. NVV-SuperBench: 0.75 (English) / 0.85 (Chinese). Full table in docs/deploy.md. This build speaks Chinese and English with one built-in voice; for voice cloning or more languages use Tag or Flash.

E2E can be served in two ways, with the same weights and the same voice: by our own engine (docs/deploy.md, the numbers above), or as one vLLM-Omni model like Tag and Flash (e2e/: /v1/realtime and /v1/chat/completions; first audio about 0.35 s for one user and 0.58–0.65 s (median) for 16). Both are measured on an H100. The E2E voice stages run about 2× slower on an RTX PRO 6000 (about 4–8 users per GPU); on such cards use Tag or Flash if you need more users per GPU.

Fine-tuning

Adapt the brain (persona, tone, domain knowledge, tool use) with LoRA in ms-swift on one 80 GB GPU. The voice, expressive sound tags and turn-taking are kept. See finetune/.

From the same team

License

Apache-2.0 for code and weights. See each build's NOTICE for third-party components.

Sophon series · by Adventists AI

sophon-realtime (adventists-ai/sophon-realtime)

正在搜索数据库…

这是什么?这是一个开源的全双工语音模型系列:你说话时它也在听,能听出你的语气,并用会笑、会叹气、会换气的声音回答你。
它是模型、服务器还是应用?既是模型,也是服务器:由 32B 或 4B 的“大脑”生成每条回复,再由语音环节把回复念出来。你可以自己部署成实时接口,也可以在 Mac 上用 MLX 在本地运行。
我该怎么把它跑起来?请参考 tag/README.md,在一块 80 GB 的 GPU 上运行 Tag 版本的一键安装脚本。请保留 [trt] 这个额外选项,因为没有 TensorRT 的话,高负载下大约慢 3 倍。如果没有 GPU,请改用托管 API。
有什么需要注意的坑吗?4B 版本的“大脑”保留了对话能力,但知识量少得多(VoiceBench 得分 62.4,32B 版本为 82.8)。如果回答必须准确,请使用 32B 模型或托管 API。
Mac 上也能跑吗?可以:mac/ 版本借助 MLX 在 Apple 芯片上本地运行 Flash-4bit,支持轮流发言、打断插话和回声消除。你停止说话后约 1 秒,它就开始回答。
我的语音数据会传到哪里?如果自己部署,数据就留在你的服务器上。Mac 版本完全在你的电脑上离线运行。托管 API 则是一项独立的云服务。
它和普通的语音处理流程有什么不同?它不会等整轮话说完才开始。它会持续聆听,在你第一次停顿时就起草回复;如果你还在说,就丢弃草稿;你打断时它会立即停止,因此对话更自然流畅。
笑声、叹气这类声音标签到底有什么作用?“大脑”会在回复中写入 [laughter]、[sigh] 之类的标记,表达型语音模型再把它们演绎成声音,而不是把它们读出来。
我能把现有的 OpenAI Realtime 客户端接到它上面吗?可以:各个版本都遵循 OpenAI Realtime 协议。在 Live 模式下,GPT-Live 客户端只需更改 URL 和密钥即可切换,而查询请求会转发到你自己的后端。
有没有适合消费级显卡的版本?有:4B 版本可以在一块 24 GB(Flash-4B)或 32 GB(Tag-4B、E2E-4B)的 GPU 上运行。它们保留了相同的轮流发言能力和声音标签,但知识量比 32B 模型少得多。
大小
7.2 MB
星标
436
时长
14天
提交 · 90 天