The project's README from github.com/adventists-ai/sophon-realtime. As one markdown file: README.md.
English | 中文
Sophon-Realtime v1 is an open full-duplex voice model family: it listens while you talk, hears how you say things, and answers with a voice that laughs, sighs and breathes when the moment calls for it.
Real conversations with Sophon-Realtime, recorded as they happened: no edits, no speed-up. Turn the sound on.
https://github.com/user-attachments/assets/4d6c74f5-3bbf-4dc2-8b11-f90ff85de4a2
Self-run unless noted. VoiceBench: official evaluators with a gpt-4o-mini judge; AlpacaEval on the standard 199-question subset. Big Bench Audio: GPT-4.1 judges the transcript of the spoken answer; other models from Artificial Analysis. Full-Duplex-Bench v3: best of several runs; other systems from the FDB v3 paper (Apr 2026). Paralinguistic understanding: same prompts and option rotation for every model. NVV-SuperBench: official prompts and metric, 50 items per class per language, Gemini 3 Flash judge; the three builds speak the same tagged reply texts in the Chelsie voice. Latency and concurrency: one H100 80 GB, users arriving within 10 s.
Every release, including fixes: CHANGELOG · GitHub Releases
All three builds share the same brain: one 32B model that listens to your voice, decides when you have finished, and writes the reply. They differ only in how the reply becomes sound.
| Tag (recommended) | Flash | E2E | |
|---|---|---|---|
| How it speaks | The brain writes the reply with sound tags ([laughter], [sigh], <strong>…</strong> …); an expressive speech model performs them |
Same tagged reply, spoken by a lighter, faster speech model (one forward pass per 80 ms frame) | No speech-token model: a compact talker (about 350M parameters) reads the tagged reply as it is written and generates continuous voice vectors directly; a decoder turns them into sound |
| Best for | Expressive voice: companions, characters, demos | Speed and scale: customer service, in-car, many users per GPU | Expressive voice with the fastest first audio; our own engine, easy to modify |
| Non-verbal sounds (NVV-SuperBench, en / zh) | 0.80 / 0.86 | 0.63 / 0.77 | 0.75 / 0.85 |
| First audio, one user | ~0.6 s | 0.31 s | ~0.25 s |
| Users per 80 GB GPU (recommended) | 16 | 40 | 16 |
| Zero-shot voice cloning | yes, from a 3–10 s clip (how) | yes, from a 3–10 s clip (how) | yes, from a 5–12 s clip (how) |
| GPUs | 1 × 80 GB | 1 × 80 GB | 1 × 80 GB |
| Serving | vLLM-Omni, one command | vLLM-Omni, one command | own engine or vLLM-Omni, one command |
| Weights | HF · ModelScope | HF · ModelScope | HF · ModelScope |
| Code | tag/ |
flash/ |
own engine: docs/deploy.md · vLLM-Omni: e2e/ |
Which one? Start with Tag if you want the voice to act. Pick Flash if the number of users per GPU matters most. Pick E2E for the fastest first audio with a voice that still laughs and sighs. Tag and Flash share the same APIs, so you can switch later; E2E runs on its own engine or, since 1.1.4, on vLLM-Omni with the same APIs as Tag and Flash.
The same three builds with a 4B brain (DuplexJev-4B-Para with a tag-writing adapter), for one consumer GPU. The speech stages and voices are the same as in the 32B builds; there is no image input.
| Tag-4B | Flash-4B | E2E-4B | |
|---|---|---|---|
| GPU | one 32 GB (uses ~27 GB) | one 24 GB (uses ~21 GB) | one 32 GB (uses ~31 GB) |
| Full-Duplex-Bench v1.0 (4-task avg; 32B: 94.0) | 94.3 | 87.7 | 92.3 |
| First audio, one user (RTX PRO 6000) | 0.40 s | 0.21 s | 0.31 s |
| Install | SIZE=4B bash install.sh |
bash install.sh + weights |
SIZE=4B bash install.sh |
| Weights | HF · ModelScope | HF · ModelScope | HF · ModelScope |
The 4B brain keeps the conversation skills (turn-taking, back-channels, barge-in, sound tags at the same rate as the reference data) but knows and reasons much less: VoiceBench 62.4 (32B: 82.8), Big Bench Audio 44.3% (32B: 81.8%). Use it for local demos, companions and characters, and turn-taking research; use 32B or the hosted API when answers must be right. start.sh sees a 4B model and picks its config.
Talk to it in your own language. Sophon-Realtime listens and answers in the language you speak.
| Language | Status |
|---|---|
| Chinese, English | Fully supported |
| Italian, German, Spanish, French, Portuguese | Ready for conversation: clear speech, 80–92% of basic questions answered correctly, multi-turn quality close to English |
| Japanese, Russian | Available on the Flash build |
| Korean and more | Coming soon |
Internal tests, November 2026: 15 languages × 20 spoken questions, plus multi-turn dialogues from Omnilingua-AudioMC in 8 languages (Portuguese not included). Japanese is a priority language for us, and upcoming releases will keep improving it.
No GPU? Use the hosted API: sign up at api.adventists.ai/console and get 3 hours of free voice chat (counted on Flash). It speaks the OpenAI Realtime protocol at wss://api.adventists.ai/sophon/v1/realtime (HTTP: https://api.adventists.ai/sophon/v1); set model to sophon-realtime-tag, sophon-realtime-flash or sophon-realtime-e2e. A complete Python example is in the API docs.
Quickest (Tag build, one 80 GB GPU):
pip install "sophon-realtime[trt]" # Python 3.12, Linux, CUDA 13; installs vLLM-Omni, the Tag plugin and TensorRT
sophon-realtime serve # downloads Sophon-Realtime-32B-Tag (~42 GB) on first run, then serves on port 8000
Keep the
[trt]. Without TensorRT the voice runs in plain PyTorch and is about 3× slower under load (16 users: first audio ~2.4 s instead of ~0.75 s, with 1–1.5 s of stutter).sophon-realtime servenow warns at start-up when TensorRT is missing.
Or use the one-command scripts, which set up an isolated environment:
Tag (one 80 GB GPU):
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh # venv + vLLM-Omni + plugin, downloads the weights (~45 GB)
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en
Flash (one 80 GB GPU):
cd sophon-realtime/flash
bash install.sh
modelscope download --model AdventistsAI/Sophon-Realtime-32B-Flash --local_dir ./model # or: hf download adventists-ai/Sophon-Realtime-32B-Flash --local-dir ./model
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en --voice chelsie
Both serve ws://HOST:PORT/v1/realtime (full duplex, OpenAI-Realtime-style) and POST /v1/chat/completions (audio in, text + audio out). Details: tag/README.md · flash/README.md.
Mac (Apple silicon, 36 GB+ unified memory recommended; runs locally with MLX):
cd sophon-realtime && bash mac/install.sh # environment + Sophon-Realtime-32B-Flash-4bit-MLX (~24 GB)
~/sophon-mac/sophon --lang en # talk through your microphone
Details: mac/README.md.
Optional: Live mode (OpenAI GPT-Live compatible, see live/), next to a running engine:
cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh # GPT-Live clients: ws://HOST:8780/v1/live/sessions
live/ is a gateway that speaks the OpenAI GPT-Live protocol in front of a Tag or Flash engine: GPT-Live clients switch by changing the URL and key. Sophon keeps the voice and hands lookups and actions to your backend through delegation, then speaks the result. Optionally, a larger model can run the back office.
Give Sophon a few seconds of someone's voice and it talks to you in that voice, in real time. No fine-tuning, no restart, one GPU. Works in the Tag and Flash builds, and in E2E since 1.3.0.
https://github.com/user-attachments/assets/7f0627b0-fae4-4633-b412-88cb24942ab8
pip install -U "sophon-realtime[trt]"
sophon-realtime download --source modelscope # or Hugging Face (default)
cp alice.wav model/voices/ # 3-10 s, one speaker
sophon-realtime serve # then set "voice": "alice"
New, replaced and deleted voices take effect on the next request. Barge-in, backchannels and laughs still work in a cloned voice, and a voice cloned from one language can speak another. Everything stays on your server (details). The video also shows Live mode: Sophon hands lookups (weather, news) to your own backend, keeps talking, then says the answer.
Please only clone your own voice or voices you have permission to use. Do not use it to impersonate anyone or to deceive.
https://github.com/user-attachments/assets/74bd6096-b6fc-4433-a1d7-d3e967107193
The E2E build has no speech-token model in the middle. A compact talker (about 350M parameters) reads the brain's tagged reply while it is still being written and generates a stream of continuous voice vectors, about 6.5 frames per second; a 2-step decoder and a vocoder turn them into 24 kHz audio. Laughs, coughs, sighs, breaths and stressed words are produced by the talker itself, as part of the same stream. Everything runs on one 80 GB GPU with our own engine: docs/deploy.md. Weights: HF · ModelScope.
Measured from a fresh install on one H100: first audio for one user in about 0.25 s; 16 users arriving within 10 s get first audio in 0.44 s (median) / 0.56 s (P90) with no reply falling behind playback. NVV-SuperBench: 0.75 (English) / 0.85 (Chinese). Full table in docs/deploy.md. This build speaks Chinese and English with one built-in voice; for voice cloning or more languages use Tag or Flash.
E2E can be served in two ways, with the same weights and the same voice: by our own engine (docs/deploy.md, the numbers above), or as one vLLM-Omni model like Tag and Flash (e2e/: /v1/realtime and /v1/chat/completions; first audio about 0.35 s for one user and 0.58–0.65 s (median) for 16). Both are measured on an H100. The E2E voice stages run about 2× slower on an RTX PRO 6000 (about 4–8 users per GPU); on such cards use Tag or Flash if you need more users per GPU.
Adapt the brain (persona, tone, domain knowledge, tool use) with LoRA in ms-swift on one 80 GB GPU. The voice, expressive sound tags and turn-taking are kept. See finetune/.
Apache-2.0 for code and weights. See each build's NOTICE for third-party components.
Sophon series · by Adventists AI