speech-swift README

The project's README from github.com/soniqo/speech-swift. As one markdown file: README.md.

Speech Swift

AI speech models for Apple Silicon, powered by MLX Swift and CoreML.

📖 Read in: English · 中文 · 日本語 · 한국어 · Español · Deutsch · Français · हिन्दी · Português · Русский · العربية · Tiếng Việt · Türkçe · ไทย

On-device speech recognition, synthesis, and understanding for Mac and iOS. Runs locally on Apple Silicon — no cloud, no API keys, no data leaves your device.

📚 Full Documentation → · 🤗 HuggingFace Models · 📝 Blog · 💬 Discord

Homebrew installs Verified public repositories: 15

soniqo%2Fspeech-swift | Trendshift

Local Speech AI on a MacBook — watch the 4-minute open-source library tour on YouTube

Local Speech AI on a MacBook — watch the 4-minute open-source library tour on YouTube

Use cases: Voice Agents · Transcription · Speech Generation

Built with Speech Swift

16 public repositories with verifiable Speech Swift package references.

AnythingLLM · Palmier Pro · Anarlog · ClawdHome · Jabber · Ora · VoxFlow · LokalBot · Voicey · HushType · DexDictate macOS · Watchtower · Wishper App · FriSpeak · Scribe · VoicePen

Capability groups: STT / ASR · Alignment · TTS · LLMs & translation · Speech-to-speech · Enhancement/restoration · Source separation · Music/audio generation · Wake word, VAD, diarization & speaker identity

STT / ASR

Alignment

TTS / Speech Generation

LLMs & Translation

Speech-to-Speech & Voice Agents

Enhancement, Separation & Audio Generation

Turn Detection, Diarization & Speaker Identity

Papers: Qwen3-ASR (Alibaba) · Qwen3-TTS (Alibaba) · Omnilingual ASR (Meta) · Parakeet TDT (NVIDIA) · CosyVoice 3 (Alibaba) · Kokoro (StyleTTS 2) · PersonaPlex (NVIDIA) · Mimi (Kyutai) · Hibiki (Kyutai) · Sortformer (NVIDIA)

On-device iPhone 16 Pro CoreML benchmarks (RTF, tokens/s, peak memory): docs/benchmarks/ios-coreml.md.

News

Quick start

Add the package to your Package.swift:

.package(url: "https://github.com/soniqo/speech-swift", branch: "main")

Import only the modules you need — every model is its own SPM library, so you don't pay for what you don't use:

.product(name: "ParakeetStreamingASR", package: "speech-swift"),
.product(name: "SpeechUI",             package: "speech-swift"),  // optional SwiftUI views

Transcribe an audio buffer in 3 lines:

import ParakeetStreamingASR

let model = try await ParakeetStreamingASRModel.fromPretrained()
let text = try model.transcribeAudio(audioSamples, sampleRate: 16000)

Live streaming with partials:

for await partial in model.transcribeStream(audio: samples, sampleRate: 16000) {
    print(partial.isFinal ? "FINAL: \(partial.text)" : "... \(partial.text)")
}

SwiftUI dictation view in ~10 lines:

import SwiftUI
import ParakeetStreamingASR
import SpeechUI

@MainActor
struct DictateView: View {
    @State private var store = TranscriptionStore()

    var body: some View {
        TranscriptionView(finals: store.finalLines, currentPartial: store.currentPartial)
            .task {
                let model = try? await ParakeetStreamingASRModel.fromPretrained()
                guard let model else { return }
                for await p in model.transcribeStream(audio: samples, sampleRate: 16000) {
                    store.apply(text: p.text, isFinal: p.isFinal)
                }
            }
    }
}

SpeechUI ships only TranscriptionView (finals + partials) and TranscriptionStore (streaming ASR adapter). Use AVFoundation for audio visualization and playback.

Available SPM products: Qwen3ASR, WhisperASR, MossTranscribe, Qwen3TTS, Qwen3TTSCoreML, ParakeetASR, ParakeetStreamingASR, NemotronStreamingASR, OmnilingualASR, CohereTranscribeASR, KyutaiSTT, VoxtralASR, KokoroTTS, SupertonicTTS, VibeVoiceTTS, CosyVoiceTTS, VoxCPM2TTS, IndexTTS2TTS, F5TTS, HiggsTTS, ChatterboxTTS, OmniVoiceTTS, IndicMioTTS, FishAudioTTS, MagpieTTS, MagpieTTSCoreML, MAGNeTMusicGen, StableAudio3MusicGen, FlashSR, PersonaPlex, VoiceChat, CSM, Audio2Face3D, HibikiTranslate, MADLADTranslation, GLiNER, Clef, SpeechVAD, SpeechLanguageID, SpeechWakeWord, SpeechEnhancement, SpeechRestoration, SourceSeparation, Qwen3Chat, FunctionGemma, SpeechCore, SpeechUI, AudioCommon.

Models

Compact view below. Full model catalogue with sizes, quantisations, download URLs, and memory tables → soniqo.audio/architecture.

Model Task Backends Sizes Languages
Qwen3-ASR Speech → Text MLX, CoreML (hybrid) 0.6B, 1.7B 52
WhisperASR Speech → Text CoreML (ANE) Large-v3 Turbo Multi
MOSS Transcribe Diarize Speech → Text + speaker timestamps CoreML / MLX 0.9B (MLX INT5/INT8; CoreML INT8/FP16) Multi
Parakeet TDT Speech → Text CoreML (ANE) 0.6B 25 European
Parakeet EOU Speech → Text (streaming) CoreML (ANE) 120M 25 European
Nemotron Streaming (Multilingual) Speech → Text (streaming, punctuated) CoreML (ANE), MLX 0.6B 40
Nemotron Streaming (English) Speech → Text (streaming, punctuated) CoreML (ANE) 0.6B EN
Omnilingual ASR Speech → Text CoreML (ANE), MLX 300M / 1B / 3B / 7B 1,672
Cohere Transcribe 2B Speech → Text MLX 2B (FP16 / INT5 / INT8) 14
Kyutai STT Speech → Text (streaming) MLX 1B (FP16 / INT4 / INT8) EN/FR
Voxtral Mini 3B Speech → Text MLX 3B (FP16 / INT5 / INT8) 8
Qwen3-ForcedAligner Audio + Text → Timestamps MLX, CoreML 0.6B Multi
Qwen3-TTS Text → Speech MLX, CoreML 0.6B, 1.7B 10
CosyVoice3 Text → Speech MLX 0.5B 9
VoxCPM2 Text → Speech (48 kHz, voice design + cloning) MLX 2B (bf16/int8) 30
IndexTTS2 Text → Speech (zero-shot voice cloning) MLX 1.5B-class (fp16) EN/ZH
F5-TTS Text → Speech (zero-shot voice cloning) MLX 336M (fp16) EN/ZH
Higgs TTS 3 Text → Speech (conversational, zero-shot voice cloning) MLX 4B (bf16) 100+
Kokoro-82M Text → Speech CoreML (ANE) 82M 10
Supertonic-3 Text → Speech (44.1 kHz, flow-matching, G2P-free) CoreML (ANE) 99M 31
VibeVoice Realtime-0.5B Text → Speech (long-form, multi-speaker) MLX 0.5B EN/ZH
VibeVoice 1.5B Text → Speech (up to 90-min podcast) MLX 1.5B EN/ZH
Magpie-TTS Multilingual Text → Speech (5 baked speakers, streaming) MLX / CoreML 357M (MLX INT8, CoreML INT8) 9 (CoreML excludes JA)
Chatterbox Multilingual Text → Speech (zero-shot cloning) MLX 0.8B (fp16) 23 (HE requires niqqud)
Chatterbox Flash Text → Speech (voice cloning with external reference conditioning) CoreML + MLX conditioning bridge 0.8B (fp16 Core ML) EN
OmniVoice Text → Speech (NAR diffusion, zero-shot cloning) MLX 0.8B (fp16 default / int8) 600+
Indic-Mio Text → Speech (Hindi/Indic, emotion tags, voice cloning) MLX fp16 Hindi / Indic
Fish Audio S2 Pro Text → Speech (zero-shot cloning, explicit style markers) MLX 0.5B-class (fp16) Multilingual
CSM Text → Speech (conversational, zero-shot voice cloning) MLX 1B (int8 / fp16) EN
Qwen3.5 Chat Text → Text (LLM) MLX, CoreML 0.8B Multi
Qwen3 Dense Chat Text → Text (LLM) MLX 4B Multi
Gemma 4 Chat Text → Text (LLM) MLX E2B / E4B (4-bit) Multi
FunctionGemma Text → Tool calls (LLM) CoreML 270M EN-tuned
MADLAD-400 Text → Text (Translation) MLX 3B 400+
GLiNER2.5-Decide Text → Labels + spans (classification, entity extraction) MLX 340M EN
Clef-flash Text → Probabilities for allowed answers MLX 9B (4-bit) EN-tested
Hibiki Zero-3B Speech → Speech (Translation) MLX 3B FR/ES/PT/DE → EN
PersonaPlex Speech → Speech MLX 7B EN
VoiceChat 11B Speech → Speech + Text MLX 11B (INT5 / INT8) EN
Audio2Face-3D Speech → Facial animation MLX v2.3 Mark Agnostic
Silero VAD Voice Activity Detection MLX, CoreML 309K Agnostic
Smart Turn v3.2 End-of-Turn Detection CoreML 8M 23 languages
KWS Zipformer Audio → Wake word CoreML (ANE) 3M EN/custom keywords
Pyannote VAD + Diarization MLX 1.5M Agnostic
Pyannote Community-1 Diarization + speaker embeddings CoreML (ANE) + Swift VBx 8.35M Agnostic
Sortformer Diarization (E2E), incremental streaming CoreML (ANE) 117M Agnostic
Nemotron 3 Diarization (E2E, up to 8 speakers) CoreML / MLX (INT8) 100M Agnostic
Ultra-Sortformer 8spk Diarization (E2E, up to 8 speakers, experimental) CoreML (ANE) 117M Agnostic
DeepFilterNet3 Speech Enhancement CoreML / MLX 2.1M Agnostic
LocalVQE v1.4-AEC Acoustic Echo Cancellation CoreML + C++ 200K + 2,742 Agnostic
Sidon Speech Restoration (denoise + dereverb, 48 kHz) CoreML w2v-BERT 2.0 + DAC (fp16/int8) Agnostic
HTDemucs (Demucs v4) Source Separation MLX 168M Agnostic
Open-Unmix Source Separation MLX 8.6M Agnostic
MAGNeT Text → Music (30s @ 32 kHz) MLX 300M / 1.5B (int4/int8) EN prompts
Stable Audio 3 Text → Music/audio (44.1 kHz stereo) MLX Medium 1.4B (int4/int8) EN prompts
FlashSR Audio super-resolution (48 kHz) MLX 363 MB / 720 MB (int4/int8) Agnostic
WeSpeaker Speaker Embedding MLX, CoreML 6.6M Agnostic
SpeechBrain ECAPA VoxLingua107 Language Identification MLX, CoreML 21.25M 107 languages
ReDimNet2-B6 Named Voice Identity CoreML 12.3M Agnostic

Installation

Homebrew

Requires native ARM Homebrew (/opt/homebrew). Rosetta/x86_64 Homebrew is not supported.

brew install speech

Then:

speech transcribe recording.wav
speech transcribe recording.wav --engine cohere
speech transcribe recording.wav --engine voxtral
speech transcribe recording.wav --engine moss
speech transcribe meeting.wav --engine moss --backend mlx
speech speak "Hello world"
speech csm "Nice to meet you" --ref-audio voice.wav --ref-text "reference transcript"
speech translate "Hello, how are you?" --to es
speech gliner classify "Remind me to call Dad at six PM." --labels create_reminder,send_message,other
speech clef decide request.json
speech respond --input question.wav --transcript
speech voice-chat
speech-server --port 8080            # local HTTP / WebSocket server (OpenAI-compatible /v1/realtime + /v1/audio/transcriptions)

Full CLI reference →

Swift Package Manager

dependencies: [
    .package(url: "https://github.com/soniqo/speech-swift", branch: "main")
]

Import only what you need — every model is its own SPM target:

import Qwen3ASR             // Speech recognition (MLX)
import WhisperASR           // Whisper Large-v3 Turbo (CoreML)
import MossTranscribe       // MOSS transcription with timestamps + speaker labels (CoreML + MLX)
import ParakeetASR          // Speech recognition (CoreML, batch)
import ParakeetStreamingASR // Streaming dictation with partials + EOU
import NemotronStreamingASR // Multilingual streaming ASR with native punctuation (0.6B, 40 langs)
import OmnilingualASR       // 1,672 languages (CoreML + MLX)
import CohereTranscribeASR  // Cohere Transcribe 2B (MLX, 14 languages)
import VoxtralASR           // Voxtral Mini 3B (MLX, 8 languages)
import Qwen3TTS             // Text-to-speech
import CosyVoiceTTS         // Text-to-speech with voice cloning
import VoxCPM2TTS           // 48 kHz TTS with voice cloning + voice design (2B)
import IndexTTS2TTS         // Native MLX voice cloning from reference audio
import F5TTS                // Zero-shot voice cloning (DiT flow matching + Vocos)
import HiggsTTS             // Conversational TTS + cloning (Qwen3 backbone, control tags)
import CSM                  // Conversational Speech Model — text→audio + voice cloning (Sesame CSM-1B, MLX)
import KokoroTTS            // Text-to-speech (iOS-ready)
import VibeVoiceTTS         // Long-form / multi-speaker TTS (EN/ZH)
import MagpieTTS            // Multilingual TTS (NVIDIA Magpie 357M, MLX, 9 langs)
import MagpieTTSCoreML      // Magpie CoreML backend (hybrid CoreML + MLX, 8 langs)
import FishAudioTTS         // Experimental Fish Audio S2 Pro runtime with voice cloning
import IndicMioTTS          // Hindi/Indic TTS with emotion markers
import Qwen3Chat            // On-device LLM chat
import FunctionGemma    // On-device tool-call LLM
import Clef                // Local decisions from text
import MADLADTranslation    // Many-to-many translation across 400+ languages
import HibikiTranslate      // Streaming speech-to-speech translation (FR/ES/PT/DE → EN)
import PersonaPlex          // Full-duplex speech-to-speech
import SpeechVAD            // VAD + speaker diarization + embeddings
import SpeechWakeWord       // Wake-word / keyword spotting
import SpeechEnhancement    // Noise suppression
import SpeechRestoration    // Speech restoration — denoise + dereverb (Sidon, CoreML, 48 kHz)
import SourceSeparation     // Music source separation (Open-Unmix, 4 stems)
import StableAudio3MusicGen // Text-to-audio/music generation (Stable Audio 3)
import SpeechUI             // SwiftUI components for streaming transcripts
import AudioCommon          // Shared protocols and utilities

Requirements

The macOS 15 / iOS 18 minimum comes from MLState — Apple's persistent ANE state API used by the CoreML pipelines (Qwen3-ASR, Qwen3-Chat, Qwen3-TTS) to keep KV caches resident on the Neural Engine across token steps.

Build from source

git clone https://github.com/soniqo/speech-swift
cd speech-swift
make build

make build compiles the Swift package and the MLX Metal shader library. The Metal library is required for GPU inference — without it you'll see Failed to load the default metallib at runtime. make debug for debug builds, make test for the test suite.

Full build and install guide →

Demo apps

Run the VoiceChat reminders demo after a release build:

./.build/release/speech voice-chat \
  --model /path/to/voicechat-mlx-int5 \
  --mcp-config Examples/VoiceChatMCP/apple-reminders.json

Add --debug-timeline for phrase, generated-pronunciation-end, and model-decoded tool lifecycle timestamps. It can reveal tool arguments, so keep it out of shared logs. See each app's README or the linked VoiceChat guide for build and runtime details.

Code examples

The snippets below show the minimal path for each domain. Every section links to a full guide on soniqo.audio with configuration options, multiple backends, streaming patterns, and CLI recipes.

Speech-to-Text — full guide →

import Qwen3ASR

let model = try await Qwen3ASRModel.fromPretrained()
let text = model.transcribe(audio: audioSamples, sampleRate: 16000)

Alternative backends: WhisperASR (Whisper Large-v3 Turbo, native CoreML), Parakeet TDT (CoreML, 32× realtime), Omnilingual ASR (1,672 languages, CoreML or MLX), Streaming dictation (live partials).

Forced Alignment — full guide →

import Qwen3ASR

let aligner = try await Qwen3ForcedAligner.fromPretrained()
let aligned = aligner.align(
    audio: audioSamples,
    text: "Can you guarantee that the replacement part will be shipped tomorrow?",
    sampleRate: 24000
)
for word in aligned {
    print("[\(word.startTime)s - \(word.endTime)s] \(word.text)")
}

Text-to-Speech — full guide →

import Qwen3TTS
import AudioCommon

let model = try await Qwen3TTSModel.fromPretrained()
let audio = model.synthesize(text: "Hello world", language: "english")
try WAVWriter.write(samples: audio, sampleRate: 24000, to: outputURL)

Alternative TTS engines: CosyVoice3 (streaming + voice cloning + emotion tags), Kokoro-82M (iOS-ready, 54 voices), VibeVoice (long-form podcast / multi-speaker, EN/ZH), Fish Audio S2 Pro (experimental zero-shot cloning + bracket style markers), Voice cloning.

Speech-to-Speech — full guide →

import PersonaPlex

let model = try await PersonaPlexModel.fromPretrained()
let responseAudio = model.respond(userAudio: userSamples)
// 24 kHz mono Float32 output ready for playback

LLM Chat — full guide →

import Qwen3Chat
import FunctionGemma

let chat = try await Qwen35MLXChat.fromPretrained()
chat.chat(messages: [(.user, "Explain MLX in one sentence")]) { token, isFinal in
    print(token, terminator: "")
}

Local decisions with Clef-flash

Give Clef some text and a list of answers. It scores each answer so your app can choose the next step. Supports choices, true/false questions, and ordered scores. Add the Clef SPM product to use the library.

import Clef

let model = try await Clef.fromPretrained()
let result = try model.decide(
    state: "Please turn the kitchen lights on.",
    questions: [
        ClefQuestion(id: "action", instructions: "Which action was requested?",
            kind: .choice(["on": "Turn lights on", "off": "Turn lights off", "other": "Other request"]))
    ])
print(result.decisions[0].selectedOption)
print(result.decisions[0].probabilities)

Requires Apple Silicon and macOS 15+. The first load downloads about 5.3 GB; runtime memory is additional. Keep one instance loaded and use it serially.

Model · Usage · Measured latency and memory

Translation — full guide →

import MADLADTranslation

let translator = try await MADLADTranslator.fromPretrained()
let es = try translator.translate("Hello, how are you?", to: "es")
// → "Hola, ¿cómo estás?"

Speech Translation — full guide →

import HibikiTranslate
import AudioCommon

let model = try await HibikiTranslateModel.fromPretrained()
let pcm = try AudioFileLoader.load(url: input, targetSampleRate: 24000)
let (englishAudio, textTokens) = model.translate(
    sourceAudio: pcm, sourceLanguage: .fr
)
// Hibiki Zero-3B — FR/ES/PT/DE → EN, on-device, streaming Mimi codec

Voice Activity Detection — full guide →

import SpeechVAD

let vad = try await SileroVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for s in segments { print("\(s.startTime)s → \(s.endTime)s") }

Speaker Diarization — full guide →

import SpeechVAD

let diarizer = try await DiarizationPipeline.fromPretrained()
let segments = diarizer.diarize(audio: samples, sampleRate: 16000)
for s in segments { print("Speaker \(s.speakerId): \(s.startTime)s - \(s.endTime)s") }

Speech Enhancement — full guide →

import SpeechEnhancement

let denoiser = try await SpeechEnhancer.fromPretrained()             // CoreML (Neural Engine)
// let denoiser = try await SpeechEnhancer.fromPretrained(engine: .mlx)  // MLX (GPU, fp32)
let clean = try denoiser.enhance(audio: noisySamples, sampleRate: 48000)

Acoustic Echo Cancellation — full guide →

import SpeechEnhancement

let aec = try await LocalVQEEchoCanceller.fromPretrained()
let cleanMicrophone = try aec.processFrame(
    microphone: microphoneFrame,
    reference: playbackReferenceFrame
)

Speech Restoration — full guide →

Joint denoise and dereverb with Sidon (w2v-BERT 2.0 predictor + DAC vocoder, Core ML). Unlike a generic noise suppressor, Sidon is trained to preserve speaker identity, so it is well suited to cleaning a noisy or reverberant voice-cloning reference before TTS. Input is 16 kHz; output is 48 kHz mono.

import SpeechRestoration

let restorer = try await SpeechRestorer.fromPretrained()          // .fp16 (default) or .int8
let clean = try restorer.restore(audio: noisySamples, sampleRate: 16000)  // → 48 kHz

From the CLI:

speech restore noisy.wav -o clean.wav            # denoise + dereverb, 48 kHz output
speech restore noisy.wav --variant int8          # smaller, lower peak RAM

# Clean a voice-cloning reference before TTS (opt-in; preserves speaker identity):
speech speak "Hello" --engine voxcpm2 --voice-sample ref.wav --clean-reference

Voice Pipeline (ASR → LLM → TTS) — full guide →

import SpeechCore

let pipeline = VoicePipeline(
    stt: parakeetASR,
    tts: qwen3TTS,
    vad: sileroVAD,
    config: .init(mode: .voicePipeline),
    onEvent: { event in print(event) }
)
pipeline.start()
pipeline.pushAudio(micSamples)

VoicePipeline is the real-time voice-agent state machine (powered by speech-core) with VAD-driven turn detection, interruption handling, and eager STT. It connects any SpeechRecognitionModel + SpeechGenerationModel + StreamingVADProvider.

HTTP API server

speech-server --port 8080

Exposes every model via HTTP REST + WebSocket endpoints, including OpenAI-compatible APIs: a Realtime WebSocket at /v1/realtime and a transcription REST endpoint at /v1/audio/transcriptions. Realtime sessions use explicit input_audio_buffer.commit by default; send session.update with turn_detection.type set to server_vad for Silero-based automatic speech start/end detection, prefix padding, and bounded turn length. See Sources/AudioServer/ and Shared Protocols.

Papers

Architecture

speech-swift is split into one SPM target per model so consumers only pay for what they import. Shared infrastructure lives in AudioCommon (protocols, audio I/O, HuggingFace downloader, SentencePieceModel) and MLXCommon (weight loading, QuantizedLinear helpers, SDPA multi-head attention helper).

Full architecture diagram with backends, memory tables, and module map → soniqo.audio/architecture · API reference → soniqo.audio/api · Benchmarks → soniqo.audio/benchmarks

Local docs (repo):

Cache configuration

Model weights download from HuggingFace on first use and cache to ~/Library/Caches/qwen3-speech/. Override with QWEN3_CACHE_DIR (CLI) or cacheDir: (Swift API). All fromPretrained() entry points also accept offlineMode: true to skip network when weights are already cached.

Users in mainland China (or anywhere huggingface.co is slow/blocked) can fetch from a mirror by setting HF_ENDPOINT, e.g. export HF_ENDPOINT=https://hf-mirror.com.

See docs/inference/cache-and-offline.md for full details including sandboxed iOS container paths.

MLX Metal library

If you see Failed to load the default metallib at runtime, the Metal shader library is missing. Run make build or ./scripts/build_mlx_metallib.sh release after a manual swift build. If the Metal Toolchain is missing, install it first:

xcodebuild -downloadComponent MetalToolchain

Testing

make test                            # full suite (unit + E2E with model downloads)
swift test --skip E2E                # unit only (CI-safe, no downloads)
swift test --filter Qwen3ASRTests    # specific module

E2E test classes use the E2E prefix so CI can filter them out with --skip E2E. See CLAUDE.md for the full testing convention.

Contributing

PRs welcome — bug fixes, new model integrations, documentation. Fork, create a feature branch, make build && make test, open a PR against main.

License

Apache 2.0

speech-swift (soniqo/speech-swift)

Searching Database…

1.2K ·GitHub
Speech-swift

Local Speech AI on a MacBook — watch the 4-minute open-source library tour on YouTube

Creator
soniqo
Size
14.2 MB
Stars
1.2K
Updated
CommitOct 17Releasev0.0.28Sep 24
Age
8mo
Commits · 90 Days