sophon-realtime README

The project's README from github.com/adventists-ai/sophon-realtime. As one markdown file: README.md.

Sophon-Realtime v1: hears your tone, speaks with one

English  |  中文

Hears your tone. Speaks with one.

Sophon-Realtime v1 is an open full-duplex voice model family: it listens while you talk, hears how you say things, and answers with a voice that laughs, sighs and breathes when the moment calls for it.

Tag on Hugging Face Flash on Hugging Face Tag on ModelScope Flash on ModelScope Try it online Apache-2.0

Demo

Real conversations with Sophon-Realtime, recorded as they happened: no edits, no speed-up. Turn the sound on.

https://github.com/user-attachments/assets/4d6c74f5-3bbf-4dc2-8b11-f90ff85de4a2

Chinese demo

https://github.com/user-attachments/assets/1d9442ef-84b0-45a3-acd4-2d05a9562397

Highlights

VoiceBench overall, full-duplex models Big Bench Audio

Full-Duplex-Bench v3 tool use, Pass@1 Paralinguistic understanding, public benchmarks combined

NVV-SuperBench, non-verbal sounds in speech, three builds

Self-run unless noted. VoiceBench: official evaluators with a gpt-4o-mini judge; AlpacaEval on the standard 199-question subset. Big Bench Audio: GPT-4.1 judges the transcript of the spoken answer; other models from Artificial Analysis. Full-Duplex-Bench v3: best of several runs; other systems from the FDB v3 paper (Apr 2026). Paralinguistic understanding: same prompts and option rotation for every model. NVV-SuperBench: official prompts and metric, 50 items per class per language, Gemini 3 Flash judge; the three builds speak the same tagged reply texts in the Chelsie voice. Latency and concurrency: one H100 80 GB, users arriving within 10 s.

News

Every release, including fixes: CHANGELOG · GitHub Releases

Three builds, one brain

All three builds share the same brain: one 32B model that listens to your voice, decides when you have finished, and writes the reply. They differ only in how the reply becomes sound.

Tag (recommended) Flash E2E
How it speaks The brain writes the reply with sound tags ([laughter], [sigh], <strong>…</strong> …); an expressive speech model performs them Same tagged reply, spoken by a lighter, faster speech model (one forward pass per 80 ms frame) No speech-token model: a compact talker (about 350M parameters) reads the tagged reply as it is written and generates continuous voice vectors directly; a decoder turns them into sound
Best for Expressive voice: companions, characters, demos Speed and scale: customer service, in-car, many users per GPU Expressive voice with the fastest first audio; our own engine, easy to modify
Non-verbal sounds (NVV-SuperBench, en / zh) 0.80 / 0.86 0.63 / 0.77 0.75 / 0.85
First audio, one user ~0.6 s 0.31 s ~0.25 s
Users per 80 GB GPU (recommended) 16 40 16
Zero-shot voice cloning yes, from a 3–10 s clip (how) yes, from a 3–10 s clip (how) yes, from a 5–12 s clip (how)
GPUs 1 × 80 GB 1 × 80 GB 1 × 80 GB
Serving vLLM-Omni, one command vLLM-Omni, one command own engine or vLLM-Omni, one command
Weights HF · ModelScope HF · ModelScope HF · ModelScope
Code tag/ flash/ own engine: docs/deploy.md · vLLM-Omni: e2e/

Which one? Start with Tag if you want the voice to act. Pick Flash if the number of users per GPU matters most. Pick E2E for the fastest first audio with a voice that still laughs and sighs. Tag and Flash share the same APIs, so you can switch later; E2E runs on its own engine or, since 1.1.4, on vLLM-Omni with the same APIs as Tag and Flash.

4B builds (new in 1.4)

The same three builds with a 4B brain (DuplexJev-4B-Para with a tag-writing adapter), for one consumer GPU. The speech stages and voices are the same as in the 32B builds; there is no image input.

Tag-4B Flash-4B E2E-4B
GPU one 32 GB (uses ~27 GB) one 24 GB (uses ~21 GB) one 32 GB (uses ~31 GB)
Full-Duplex-Bench v1.0 (4-task avg; 32B: 94.0) 94.3 87.7 92.3
First audio, one user (RTX PRO 6000) 0.40 s 0.21 s 0.31 s
Install SIZE=4B bash install.sh bash install.sh + weights SIZE=4B bash install.sh
Weights HF · ModelScope HF · ModelScope HF · ModelScope

The 4B brain keeps the conversation skills (turn-taking, back-channels, barge-in, sound tags at the same rate as the reference data) but knows and reasons much less: VoiceBench 62.4 (32B: 82.8), Big Bench Audio 44.3% (32B: 81.8%). Use it for local demos, companions and characters, and turn-taking research; use 32B or the hosted API when answers must be right. start.sh sees a 4B model and picks its config.

Languages

Talk to it in your own language. Sophon-Realtime listens and answers in the language you speak.

Language Status
Chinese, English Fully supported
Italian, German, Spanish, French, Portuguese Ready for conversation: clear speech, 80–92% of basic questions answered correctly, multi-turn quality close to English
Japanese, Russian Available on the Flash build
Korean and more Coming soon

Internal tests, November 2026: 15 languages × 20 spoken questions, plus multi-turn dialogues from Omnilingua-AudioMC in 8 languages (Portuguese not included). Japanese is a priority language for us, and upcoming releases will keep improving it.

How it works

Quick start

No GPU? Use the hosted API: sign up at api.adventists.ai/console and get 3 hours of free voice chat (counted on Flash). It speaks the OpenAI Realtime protocol at wss://api.adventists.ai/sophon/v1/realtime (HTTP: https://api.adventists.ai/sophon/v1); set model to sophon-realtime-tag, sophon-realtime-flash or sophon-realtime-e2e. A complete Python example is in the API docs.

Quickest (Tag build, one 80 GB GPU):

pip install "sophon-realtime[trt]"   # Python 3.12, Linux, CUDA 13; installs vLLM-Omni, the Tag plugin and TensorRT
sophon-realtime serve           # downloads Sophon-Realtime-32B-Tag (~42 GB) on first run, then serves on port 8000

Keep the [trt]. Without TensorRT the voice runs in plain PyTorch and is about 3× slower under load (16 users: first audio ~2.4 s instead of ~0.75 s, with 1–1.5 s of stutter). sophon-realtime serve now warns at start-up when TensorRT is missing.

Or use the one-command scripts, which set up an isolated environment:

Tag (one 80 GB GPU):

git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh                 # venv + vLLM-Omni + plugin, downloads the weights (~45 GB)
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en

Flash (one 80 GB GPU):

cd sophon-realtime/flash
bash install.sh
modelscope download --model AdventistsAI/Sophon-Realtime-32B-Flash --local_dir ./model   # or: hf download adventists-ai/Sophon-Realtime-32B-Flash --local-dir ./model
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en --voice chelsie

Both serve ws://HOST:PORT/v1/realtime (full duplex, OpenAI-Realtime-style) and POST /v1/chat/completions (audio in, text + audio out). Details: tag/README.md · flash/README.md.

Mac (Apple silicon, 36 GB+ unified memory recommended; runs locally with MLX):

cd sophon-realtime && bash mac/install.sh   # environment + Sophon-Realtime-32B-Flash-4bit-MLX (~24 GB)
~/sophon-mac/sophon --lang en               # talk through your microphone

Details: mac/README.md.

Optional: Live mode (OpenAI GPT-Live compatible, see live/), next to a running engine:

cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh   # GPT-Live clients: ws://HOST:8780/v1/live/sessions

Live mode (OpenAI GPT-Live compatible)

live/ is a gateway that speaks the OpenAI GPT-Live protocol in front of a Tag or Flash engine: GPT-Live clients switch by changing the URL and key. Sophon keeps the voice and hands lookups and actions to your backend through delegation, then speaks the result. Optionally, a larger model can run the back office.

Zero-shot voice cloning (new in v1.1)

Give Sophon a few seconds of someone's voice and it talks to you in that voice, in real time. No fine-tuning, no restart, one GPU. Works in the Tag and Flash builds, and in E2E since 1.3.0.

https://github.com/user-attachments/assets/7f0627b0-fae4-4633-b412-88cb24942ab8

pip install -U "sophon-realtime[trt]"
sophon-realtime download --source modelscope   # or Hugging Face (default)
cp alice.wav model/voices/                      # 3-10 s, one speaker
sophon-realtime serve                           # then set "voice": "alice"

New, replaced and deleted voices take effect on the next request. Barge-in, backchannels and laughs still work in a cloned voice, and a voice cloned from one language can speak another. Everything stays on your server (details). The video also shows Live mode: Sophon hands lookups (weather, news) to your own backend, keeps talking, then says the answer.

Please only clone your own voice or voices you have permission to use. Do not use it to impersonate anyone or to deceive.

Chinese-subtitled version

https://github.com/user-attachments/assets/74bd6096-b6fc-4433-a1d7-d3e967107193

E2E

The E2E build has no speech-token model in the middle. A compact talker (about 350M parameters) reads the brain's tagged reply while it is still being written and generates a stream of continuous voice vectors, about 6.5 frames per second; a 2-step decoder and a vocoder turn them into 24 kHz audio. Laughs, coughs, sighs, breaths and stressed words are produced by the talker itself, as part of the same stream. Everything runs on one 80 GB GPU with our own engine: docs/deploy.md. Weights: HF · ModelScope.

Measured from a fresh install on one H100: first audio for one user in about 0.25 s; 16 users arriving within 10 s get first audio in 0.44 s (median) / 0.56 s (P90) with no reply falling behind playback. NVV-SuperBench: 0.75 (English) / 0.85 (Chinese). Full table in docs/deploy.md. This build speaks Chinese and English with one built-in voice; for voice cloning or more languages use Tag or Flash.

E2E can be served in two ways, with the same weights and the same voice: by our own engine (docs/deploy.md, the numbers above), or as one vLLM-Omni model like Tag and Flash (e2e/: /v1/realtime and /v1/chat/completions; first audio about 0.35 s for one user and 0.58–0.65 s (median) for 16). Both are measured on an H100. The E2E voice stages run about 2× slower on an RTX PRO 6000 (about 4–8 users per GPU); on such cards use Tag or Flash if you need more users per GPU.

Fine-tuning

Adapt the brain (persona, tone, domain knowledge, tool use) with LoRA in ms-swift on one 80 GB GPU. The voice, expressive sound tags and turn-taking are kept. See finetune/.

From the same team

License

Apache-2.0 for code and weights. See each build's NOTICE for third-party components.

Sophon series · by Adventists AI

sophon-realtime (adventists-ai/sophon-realtime)

Searching Database…

What is this?An open-source family of full-duplex voice models: it listens while you talk, hears your tone and answers with a voice that can laugh, sigh and breathe.
Is it a model, a server or an app?Both a model and a server: a 32B or 4B brain writes each reply and a speech stage voices it. You host it yourself as a realtime endpoint, or run the Mac build locally with MLX.
How do I get it running?Follow tag/README.md and run the one-command install script for the Tag build on one 80 GB GPU. Keep the [trt] extra, because without TensorRT it runs about 3x slower under load. With no GPU, use the hosted API instead.
Any catches I should know?The 4B brain keeps the conversation skills but knows much less (VoiceBench 62.4 against 82.8 for 32B). Use the 32B model or the hosted API when answers must be right.
Does it run on a Mac too?Yes: the mac/ build runs Flash-4bit locally on Apple silicon with MLX, with turn-taking, barge-in and echo cancellation. It starts speaking about 1 s after you stop.
Where does my voice data go?If you self-host, it stays on your server. The Mac build runs fully offline on your machine. The hosted API is a separate cloud service.
What makes it different from a typical speech pipeline?It does not wait for a full turn. It listens continuously, drafts a reply from your first pause, drops the draft if you keep talking, and stops when you interrupt, so conversations flow more naturally.
What do the sound tags like laughter or sigh actually do?The brain writes marks such as [laughter] or [sigh] into its reply, and an expressive speech model performs them as sound instead of reading them aloud.
Can I connect existing OpenAI Realtime clients to it?Yes: the builds follow the OpenAI Realtime protocol. Live mode lets GPT-Live clients switch by changing the URL and key, while lookups go to your own backend.
Is there a version for a consumer GPU?Yes: the 4B builds fit one 24 GB (Flash-4B) or 32 GB (Tag-4B, E2E-4B) GPU. They keep the same turn-taking and sound tags but know much less than the 32B models.
Size
7.2 MB
Stars
436
Age
14d
Commits · 90 Days