sophon-realtime README

README проекта с github.com/adventists-ai/sophon-realtime. Одним файлом markdown: README.md.

Sophon-Realtime v1: hears your tone, speaks with one

English  |  中文

Hears your tone. Speaks with one.

Sophon-Realtime v1 is an open full-duplex voice model family: it listens while you talk, hears how you say things, and answers with a voice that laughs, sighs and breathes when the moment calls for it.

Tag on Hugging Face Flash on Hugging Face Tag on ModelScope Flash on ModelScope Try it online Apache-2.0

Demo

Real conversations with Sophon-Realtime, recorded as they happened: no edits, no speed-up. Turn the sound on.

https://github.com/user-attachments/assets/4d6c74f5-3bbf-4dc2-8b11-f90ff85de4a2

Chinese demo

https://github.com/user-attachments/assets/1d9442ef-84b0-45a3-acd4-2d05a9562397

Highlights

VoiceBench overall, full-duplex models Big Bench Audio

Full-Duplex-Bench v3 tool use, Pass@1 Paralinguistic understanding, public benchmarks combined

NVV-SuperBench, non-verbal sounds in speech, three builds

Self-run unless noted. VoiceBench: official evaluators with a gpt-4o-mini judge; AlpacaEval on the standard 199-question subset. Big Bench Audio: GPT-4.1 judges the transcript of the spoken answer; other models from Artificial Analysis. Full-Duplex-Bench v3: best of several runs; other systems from the FDB v3 paper (Apr 2026). Paralinguistic understanding: same prompts and option rotation for every model. NVV-SuperBench: official prompts and metric, 50 items per class per language, Gemini 3 Flash judge; the three builds speak the same tagged reply texts in the Chelsie voice. Latency and concurrency: one H100 80 GB, users arriving within 10 s.

News

Every release, including fixes: CHANGELOG · GitHub Releases

Three builds, one brain

All three builds share the same brain: one 32B model that listens to your voice, decides when you have finished, and writes the reply. They differ only in how the reply becomes sound.

Tag (recommended) Flash E2E
How it speaks The brain writes the reply with sound tags ([laughter], [sigh], <strong>…</strong> …); an expressive speech model performs them Same tagged reply, spoken by a lighter, faster speech model (one forward pass per 80 ms frame) No speech-token model: a compact talker (about 350M parameters) reads the tagged reply as it is written and generates continuous voice vectors directly; a decoder turns them into sound
Best for Expressive voice: companions, characters, demos Speed and scale: customer service, in-car, many users per GPU Expressive voice with the fastest first audio; our own engine, easy to modify
Non-verbal sounds (NVV-SuperBench, en / zh) 0.80 / 0.86 0.63 / 0.77 0.75 / 0.85
First audio, one user ~0.6 s 0.31 s ~0.25 s
Users per 80 GB GPU (recommended) 16 40 16
Zero-shot voice cloning yes, from a 3–10 s clip (how) yes, from a 3–10 s clip (how) yes, from a 5–12 s clip (how)
GPUs 1 × 80 GB 1 × 80 GB 1 × 80 GB
Serving vLLM-Omni, one command vLLM-Omni, one command own engine or vLLM-Omni, one command
Weights HF · ModelScope HF · ModelScope HF · ModelScope
Code tag/ flash/ own engine: docs/deploy.md · vLLM-Omni: e2e/

Which one? Start with Tag if you want the voice to act. Pick Flash if the number of users per GPU matters most. Pick E2E for the fastest first audio with a voice that still laughs and sighs. Tag and Flash share the same APIs, so you can switch later; E2E runs on its own engine or, since 1.1.4, on vLLM-Omni with the same APIs as Tag and Flash.

4B builds (new in 1.4)

The same three builds with a 4B brain (DuplexJev-4B-Para with a tag-writing adapter), for one consumer GPU. The speech stages and voices are the same as in the 32B builds; there is no image input.

Tag-4B Flash-4B E2E-4B
GPU one 32 GB (uses ~27 GB) one 24 GB (uses ~21 GB) one 32 GB (uses ~31 GB)
Full-Duplex-Bench v1.0 (4-task avg; 32B: 94.0) 94.3 87.7 92.3
First audio, one user (RTX PRO 6000) 0.40 s 0.21 s 0.31 s
Install SIZE=4B bash install.sh bash install.sh + weights SIZE=4B bash install.sh
Weights HF · ModelScope HF · ModelScope HF · ModelScope

The 4B brain keeps the conversation skills (turn-taking, back-channels, barge-in, sound tags at the same rate as the reference data) but knows and reasons much less: VoiceBench 62.4 (32B: 82.8), Big Bench Audio 44.3% (32B: 81.8%). Use it for local demos, companions and characters, and turn-taking research; use 32B or the hosted API when answers must be right. start.sh sees a 4B model and picks its config.

Languages

Talk to it in your own language. Sophon-Realtime listens and answers in the language you speak.

Language Status
Chinese, English Fully supported
Italian, German, Spanish, French, Portuguese Ready for conversation: clear speech, 80–92% of basic questions answered correctly, multi-turn quality close to English
Japanese, Russian Available on the Flash build
Korean and more Coming soon

Internal tests, November 2026: 15 languages × 20 spoken questions, plus multi-turn dialogues from Omnilingua-AudioMC in 8 languages (Portuguese not included). Japanese is a priority language for us, and upcoming releases will keep improving it.

How it works

Quick start

No GPU? Use the hosted API: sign up at api.adventists.ai/console and get 3 hours of free voice chat (counted on Flash). It speaks the OpenAI Realtime protocol at wss://api.adventists.ai/sophon/v1/realtime (HTTP: https://api.adventists.ai/sophon/v1); set model to sophon-realtime-tag, sophon-realtime-flash or sophon-realtime-e2e. A complete Python example is in the API docs.

Quickest (Tag build, one 80 GB GPU):

pip install "sophon-realtime[trt]"   # Python 3.12, Linux, CUDA 13; installs vLLM-Omni, the Tag plugin and TensorRT
sophon-realtime serve           # downloads Sophon-Realtime-32B-Tag (~42 GB) on first run, then serves on port 8000

Keep the [trt]. Without TensorRT the voice runs in plain PyTorch and is about 3× slower under load (16 users: first audio ~2.4 s instead of ~0.75 s, with 1–1.5 s of stutter). sophon-realtime serve now warns at start-up when TensorRT is missing.

Or use the one-command scripts, which set up an isolated environment:

Tag (one 80 GB GPU):

git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh                 # venv + vLLM-Omni + plugin, downloads the weights (~45 GB)
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en

Flash (one 80 GB GPU):

cd sophon-realtime/flash
bash install.sh
modelscope download --model AdventistsAI/Sophon-Realtime-32B-Flash --local_dir ./model   # or: hf download adventists-ai/Sophon-Realtime-32B-Flash --local-dir ./model
GPU=0 PORT=8000 bash start.sh
python client_example.py --wav my_question.wav --lang en --voice chelsie

Both serve ws://HOST:PORT/v1/realtime (full duplex, OpenAI-Realtime-style) and POST /v1/chat/completions (audio in, text + audio out). Details: tag/README.md · flash/README.md.

Mac (Apple silicon, 36 GB+ unified memory recommended; runs locally with MLX):

cd sophon-realtime && bash mac/install.sh   # environment + Sophon-Realtime-32B-Flash-4bit-MLX (~24 GB)
~/sophon-mac/sophon --lang en               # talk through your microphone

Details: mac/README.md.

Optional: Live mode (OpenAI GPT-Live compatible, see live/), next to a running engine:

cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh   # GPT-Live clients: ws://HOST:8780/v1/live/sessions

Live mode (OpenAI GPT-Live compatible)

live/ is a gateway that speaks the OpenAI GPT-Live protocol in front of a Tag or Flash engine: GPT-Live clients switch by changing the URL and key. Sophon keeps the voice and hands lookups and actions to your backend through delegation, then speaks the result. Optionally, a larger model can run the back office.

Zero-shot voice cloning (new in v1.1)

Give Sophon a few seconds of someone's voice and it talks to you in that voice, in real time. No fine-tuning, no restart, one GPU. Works in the Tag and Flash builds, and in E2E since 1.3.0.

https://github.com/user-attachments/assets/7f0627b0-fae4-4633-b412-88cb24942ab8

pip install -U "sophon-realtime[trt]"
sophon-realtime download --source modelscope   # or Hugging Face (default)
cp alice.wav model/voices/                      # 3-10 s, one speaker
sophon-realtime serve                           # then set "voice": "alice"

New, replaced and deleted voices take effect on the next request. Barge-in, backchannels and laughs still work in a cloned voice, and a voice cloned from one language can speak another. Everything stays on your server (details). The video also shows Live mode: Sophon hands lookups (weather, news) to your own backend, keeps talking, then says the answer.

Please only clone your own voice or voices you have permission to use. Do not use it to impersonate anyone or to deceive.

Chinese-subtitled version

https://github.com/user-attachments/assets/74bd6096-b6fc-4433-a1d7-d3e967107193

E2E

The E2E build has no speech-token model in the middle. A compact talker (about 350M parameters) reads the brain's tagged reply while it is still being written and generates a stream of continuous voice vectors, about 6.5 frames per second; a 2-step decoder and a vocoder turn them into 24 kHz audio. Laughs, coughs, sighs, breaths and stressed words are produced by the talker itself, as part of the same stream. Everything runs on one 80 GB GPU with our own engine: docs/deploy.md. Weights: HF · ModelScope.

Measured from a fresh install on one H100: first audio for one user in about 0.25 s; 16 users arriving within 10 s get first audio in 0.44 s (median) / 0.56 s (P90) with no reply falling behind playback. NVV-SuperBench: 0.75 (English) / 0.85 (Chinese). Full table in docs/deploy.md. This build speaks Chinese and English with one built-in voice; for voice cloning or more languages use Tag or Flash.

E2E can be served in two ways, with the same weights and the same voice: by our own engine (docs/deploy.md, the numbers above), or as one vLLM-Omni model like Tag and Flash (e2e/: /v1/realtime and /v1/chat/completions; first audio about 0.35 s for one user and 0.58–0.65 s (median) for 16). Both are measured on an H100. The E2E voice stages run about 2× slower on an RTX PRO 6000 (about 4–8 users per GPU); on such cards use Tag or Flash if you need more users per GPU.

Fine-tuning

Adapt the brain (persona, tone, domain knowledge, tool use) with LoRA in ms-swift on one 80 GB GPU. The voice, expressive sound tags and turn-taking are kept. See finetune/.

From the same team

License

Apache-2.0 for code and weights. See each build's NOTICE for third-party components.

Sophon series · by Adventists AI

sophon-realtime (adventists-ai/sophon-realtime)

Ищем в базе…

Что это такое?Открытое семейство моделей полнодуплексного голосового общения: модель слушает, пока вы говорите, распознаёт ваш тон и отвечает голосом, который умеет смеяться, вздыхать и дышать.
Это модель, сервер или приложение?И модель, и сервер: основная модель на 32B или 4B пишет каждый ответ, а речевой модуль озвучивает его. Вы разворачиваете её у себя как realtime-эндпоинт или запускаете сборку для Mac локально на MLX.
Как это запустить?Следуйте инструкциям в tag/README.md и запустите скрипт установки одной командой для сборки Tag на одном GPU с 80 ГБ памяти. Оставьте опцию [trt], потому что без TensorRT под нагрузкой работа идёт примерно в 3 раза медленнее. Если GPU нет, используйте размещённый API.
Есть какие-то подводные камни?Основная модель на 4B сохраняет навыки разговора, но знает заметно меньше (VoiceBench 62,4 против 82,8 у 32B). Когда ответы должны быть точными, используйте модель на 32B или размещённый API.
Работает ли это и на Mac?Да: сборка mac/ запускает Flash-4bit локально на Apple silicon через MLX, с управлением очерёдностью реплик, возможностью перебить и подавлением эха. Начинает говорить примерно через 1 с после того, как вы замолчите.
Куда попадают мои голосовые данные?Если вы разворачиваете модель сами, данные остаются на вашем сервере. Сборка для Mac работает полностью офлайн на вашем компьютере. Размещённый API — это отдельный облачный сервис.
Чем это отличается от обычной речевой цепочки?Оно не ждёт окончания всей реплики. Модель слушает непрерывно, набрасывает черновик ответа с первой паузы, отбрасывает его, если вы продолжаете говорить, и останавливается, когда вы перебиваете, поэтому разговор идёт естественнее.
Что на самом деле делают звуковые метки вроде смеха или вздоха?Основная модель вписывает в ответ метки вида [laughter] или [sigh], а выразительная речевая модель воспроизводит их как звук, а не зачитывает вслух.
Можно ли подключить к нему имеющиеся клиенты OpenAI Realtime?Да: сборки следуют протоколу OpenAI Realtime. В режиме Live клиенты GPT-Live переходят на него, если поменять URL и ключ, а поисковые запросы идут на ваш собственный бэкенд.
Есть ли версия для обычной видеокарты?Да: модели на 4B помещаются на один GPU с 24 ГБ (Flash-4B) или 32 ГБ (Tag-4B, E2E-4B). Они сохраняют ту же очерёдность реплик и звуковые метки, но знают заметно меньше, чем модели на 32B.
Какие есть похожие проекты?speech-swift, openspline, MOSS-TTS-Nano, Index-Translate, unsloth, VoxCPM, ai-phone, xn-ptts
Автор
adventists-ai
Размер
7.2 MB
Звёзды
436
Возраст
14 д
Коммиты · 90 дней