speech-swift README

项目的 README,来自 github.com/soniqo/speech-swift。合并为一个 markdown 文件:README.md。

Speech Swift

AI speech models for Apple Silicon, powered by MLX Swift and CoreML.

📖 Read in: English · 中文 · 日本語 · 한국어 · Español · Deutsch · Français · हिन्दी · Português · Русский · العربية · Tiếng Việt · Türkçe · ไทย

On-device speech recognition, synthesis, and understanding for Mac and iOS. Runs locally on Apple Silicon — no cloud, no API keys, no data leaves your device.

📚 Full Documentation → · 🤗 HuggingFace Models · 📝 Blog · 💬 Discord

Homebrew installs Verified public repositories: 15

soniqo%2Fspeech-swift | Trendshift

Local Speech AI on a MacBook — watch the 4-minute open-source library tour on YouTube

Local Speech AI on a MacBook — watch the 4-minute open-source library tour on YouTube

Use cases: Voice Agents · Transcription · Speech Generation

Built with Speech Swift

16 public repositories with verifiable Speech Swift package references.

AnythingLLM · Palmier Pro · Anarlog · ClawdHome · Jabber · Ora · VoxFlow · LokalBot · Voicey · HushType · DexDictate macOS · Watchtower · Wishper App · FriSpeak · Scribe · VoicePen

Capability groups: STT / ASR · Alignment · TTS · LLMs & translation · Speech-to-speech · Enhancement/restoration · Source separation · Music/audio generation · Wake word, VAD, diarization & speaker identity

STT / ASR

Alignment

TTS / Speech Generation

LLMs & Translation

Speech-to-Speech & Voice Agents

Enhancement, Separation & Audio Generation

Turn Detection, Diarization & Speaker Identity

Papers: Qwen3-ASR (Alibaba) · Qwen3-TTS (Alibaba) · Omnilingual ASR (Meta) · Parakeet TDT (NVIDIA) · CosyVoice 3 (Alibaba) · Kokoro (StyleTTS 2) · PersonaPlex (NVIDIA) · Mimi (Kyutai) · Hibiki (Kyutai) · Sortformer (NVIDIA)

On-device iPhone 16 Pro CoreML benchmarks (RTF, tokens/s, peak memory): docs/benchmarks/ios-coreml.md.

News

Quick start

Add the package to your Package.swift:

.package(url: "https://github.com/soniqo/speech-swift", branch: "main")

Import only the modules you need — every model is its own SPM library, so you don't pay for what you don't use:

.product(name: "ParakeetStreamingASR", package: "speech-swift"),
.product(name: "SpeechUI",             package: "speech-swift"),  // optional SwiftUI views

Transcribe an audio buffer in 3 lines:

import ParakeetStreamingASR

let model = try await ParakeetStreamingASRModel.fromPretrained()
let text = try model.transcribeAudio(audioSamples, sampleRate: 16000)

Live streaming with partials:

for await partial in model.transcribeStream(audio: samples, sampleRate: 16000) {
    print(partial.isFinal ? "FINAL: \(partial.text)" : "... \(partial.text)")
}

SwiftUI dictation view in ~10 lines:

import SwiftUI
import ParakeetStreamingASR
import SpeechUI

@MainActor
struct DictateView: View {
    @State private var store = TranscriptionStore()

    var body: some View {
        TranscriptionView(finals: store.finalLines, currentPartial: store.currentPartial)
            .task {
                let model = try? await ParakeetStreamingASRModel.fromPretrained()
                guard let model else { return }
                for await p in model.transcribeStream(audio: samples, sampleRate: 16000) {
                    store.apply(text: p.text, isFinal: p.isFinal)
                }
            }
    }
}

SpeechUI ships only TranscriptionView (finals + partials) and TranscriptionStore (streaming ASR adapter). Use AVFoundation for audio visualization and playback.

Available SPM products: Qwen3ASR, WhisperASR, MossTranscribe, Qwen3TTS, Qwen3TTSCoreML, ParakeetASR, ParakeetStreamingASR, NemotronStreamingASR, OmnilingualASR, CohereTranscribeASR, KyutaiSTT, VoxtralASR, KokoroTTS, SupertonicTTS, VibeVoiceTTS, CosyVoiceTTS, VoxCPM2TTS, IndexTTS2TTS, F5TTS, HiggsTTS, BreezeTTS, FireRedTTS3TTS, MossVoiceGeneratorTTS, ChatterboxTTS, OmniVoiceTTS, IndicMioTTS, FishAudioTTS, MagpieTTS, MagpieTTSCoreML, MAGNeTMusicGen, StableAudio3MusicGen, FlashSR, PersonaPlex, VoiceChat, CSM, Audio2Face3D, HibikiTranslate, MADLADTranslation, GLiNER, Clef, SpeechVAD, SpeechLanguageID, SpeechWakeWord, SpeechEnhancement, SpeechRestoration, SourceSeparation, Qwen3Chat, FunctionGemma, SpeechCore, SpeechUI, AudioCommon.

Models

Compact view below. Full model catalogue with sizes, quantisations, download URLs, and memory tables → soniqo.audio/architecture.

Model Task Backends Sizes Languages
Qwen3-ASR Speech → Text MLX, CoreML (hybrid) 0.6B, 1.7B 52
WhisperASR Speech → Text CoreML (ANE) Large-v3 Turbo Multi
MOSS Transcribe Diarize Speech → Text + speaker timestamps CoreML / MLX 0.9B (MLX INT5/INT8; CoreML INT8/FP16) Multi
Parakeet TDT Speech → Text CoreML (ANE) 0.6B 25 European
Parakeet EOU Speech → Text (streaming) CoreML (ANE) 120M 25 European
Nemotron Streaming (Multilingual) Speech → Text (streaming, punctuated) CoreML (ANE), MLX 0.6B 40
Nemotron Streaming (English) Speech → Text (streaming, punctuated) CoreML (ANE) 0.6B EN
Omnilingual ASR Speech → Text CoreML (ANE), MLX 300M / 1B / 3B / 7B 1,672
Cohere Transcribe 2B Speech → Text MLX 2B (FP16 / INT5 / INT8) 14
Kyutai STT Speech → Text (streaming) MLX 1B (FP16 / INT4 / INT8) EN/FR
Voxtral Mini 3B Speech → Text MLX 3B (FP16 / INT5 / INT8) 8
Qwen3-ForcedAligner Audio + Text → Timestamps MLX, CoreML 0.6B Multi
Qwen3-TTS Text → Speech MLX, CoreML 0.6B, 1.7B 10
CosyVoice3 Text → Speech MLX 0.5B 9
VoxCPM2 Text → Speech (48 kHz, voice design + cloning) MLX 2B (bf16/int8) 30
IndexTTS2 Text → Speech (zero-shot voice cloning) MLX 1.5B-class (fp16) EN/ZH
F5-TTS Text → Speech (zero-shot voice cloning) MLX 336M (fp16) EN/ZH
Higgs TTS 3 Text → Speech (conversational, zero-shot voice cloning) MLX 4B (bf16) 100+
Breeze TTS 2 Text → Speech MLX BF16 / INT8 EN/ZH
FireRedTTS3-Instruct Text → Speech MLX FP32 EN/ZH
MOSS-VoiceGenerator Text → Speech MLX BF16 + FP32 codec EN/ZH
Kokoro-82M Text → Speech CoreML (ANE) 82M 10
Supertonic-3 Text → Speech (44.1 kHz, flow-matching, G2P-free) CoreML (ANE) 99M 31
VibeVoice Realtime-0.5B Text → Speech (long-form, multi-speaker) MLX 0.5B EN/ZH
VibeVoice 1.5B Text → Speech (up to 90-min podcast) MLX 1.5B EN/ZH
Magpie-TTS Multilingual Text → Speech (5 baked speakers, streaming) MLX / CoreML 357M (MLX INT8, CoreML INT8) 9 (CoreML excludes JA)
Chatterbox Multilingual Text → Speech (zero-shot cloning) MLX 0.8B (fp16) 23 (HE requires niqqud)
Chatterbox Flash Text → Speech (voice cloning with external reference conditioning) CoreML + MLX conditioning bridge 0.8B (fp16 Core ML) EN
OmniVoice Text → Speech (NAR diffusion, zero-shot cloning) MLX 0.8B (fp16 default / int8) 600+
Indic-Mio Text → Speech (Hindi/Indic, emotion tags, voice cloning) MLX fp16 Hindi / Indic
Fish Audio S2 Pro Text → Speech (zero-shot cloning, explicit style markers) MLX 0.5B-class (fp16) Multilingual
CSM Text → Speech (conversational, zero-shot voice cloning) MLX 1B (int8 / fp16) EN
Qwen3.5 Chat Text → Text (LLM) MLX, CoreML 0.8B Multi
Qwen3 Dense Chat Text → Text (LLM) MLX 4B Multi
Gemma 4 Chat Text → Text (LLM) MLX E2B / E4B (4-bit) Multi
FunctionGemma Text → Tool calls (LLM) CoreML 270M EN-tuned
MADLAD-400 Text → Text (Translation) MLX 3B 400+
GLiNER2.5-Decide Text → Labels + spans (classification, entity extraction) MLX 340M EN
Clef-flash Text → Probabilities for allowed answers MLX 9B (4-bit) EN-tested
Hibiki Zero-3B Speech → Speech (Translation) MLX 3B FR/ES/PT/DE → EN
PersonaPlex Speech → Speech MLX 7B EN
VoiceChat 11B Speech → Speech + Text MLX 11B (INT5 / INT8) EN
Audio2Face-3D Speech → Facial animation MLX v2.3 Mark Agnostic
Silero VAD Voice Activity Detection MLX, CoreML 309K Agnostic
Smart Turn v3.2 End-of-Turn Detection CoreML 8M 23 languages
p99 turn-1-mini Streaming turn completion CoreML 5.3M English
VAP (otoSpeech) Two-channel speech forecasts CoreML 5.8M English
KWS Zipformer Audio → Wake word CoreML (ANE) 3M EN/custom keywords
Pyannote VAD + Diarization MLX 1.5M Agnostic
Pyannote Community-1 Diarization + speaker embeddings CoreML (ANE) + Swift VBx 8.35M Agnostic
Sortformer Diarization (E2E), incremental streaming CoreML (ANE) 117M Agnostic
Nemotron 3 Diarization (E2E, up to 8 speakers) CoreML / MLX (INT8) 100M Agnostic
Ultra-Sortformer 8spk Diarization (E2E, up to 8 speakers, experimental) CoreML (ANE) 117M Agnostic
DeepFilterNet3 Speech Enhancement CoreML / MLX 2.1M Agnostic
LocalVQE v1.4-AEC Acoustic Echo Cancellation CoreML + C++ 200K + 2,742 Agnostic
Sidon Speech Restoration (denoise + dereverb, 48 kHz) CoreML w2v-BERT 2.0 + DAC (fp16/int8) Agnostic
HTDemucs (Demucs v4) Source Separation MLX 168M Agnostic
Open-Unmix Source Separation MLX 8.6M Agnostic
MAGNeT Text → Music (30s @ 32 kHz) MLX 300M / 1.5B (int4/int8) EN prompts
Stable Audio 3 Text → Music/audio (44.1 kHz stereo) MLX Medium 1.4B (int4/int8) EN prompts
FlashSR Audio super-resolution (48 kHz) MLX 363 MB / 720 MB (int4/int8) Agnostic
WeSpeaker Speaker Embedding MLX, CoreML 6.6M Agnostic
SpeechBrain ECAPA VoxLingua107 Language Identification MLX, CoreML 21.25M 107 languages
ReDimNet2-B6 Named Voice Identity CoreML 12.3M Agnostic

Installation

Homebrew

Requires native ARM Homebrew (/opt/homebrew). Rosetta/x86_64 Homebrew is not supported.

brew install speech

Then:

speech transcribe recording.wav
speech transcribe recording.wav --engine cohere
speech transcribe recording.wav --engine voxtral
speech transcribe recording.wav --engine moss
speech transcribe meeting.wav --engine moss --backend mlx
speech speak "Hello world"
speech csm "Nice to meet you" --ref-audio voice.wav --ref-text "reference transcript"
speech translate "Hello, how are you?" --to es
speech gliner classify "Remind me to call Dad at six PM." --labels create_reminder,send_message,other
speech clef decide request.json
speech respond --input question.wav --transcript
speech voice-chat
speech-server --port 8080            # local HTTP / WebSocket server (OpenAI-compatible /v1/realtime + /v1/audio/transcriptions)

Full CLI reference →

Swift Package Manager

dependencies: [
    .package(url: "https://github.com/soniqo/speech-swift", branch: "main")
]

Import only what you need — every model is its own SPM target:

import Qwen3ASR             // Speech recognition (MLX)
import WhisperASR           // Whisper Large-v3 Turbo (CoreML)
import MossTranscribe       // MOSS transcription with timestamps + speaker labels (CoreML + MLX)
import ParakeetASR          // Speech recognition (CoreML, batch)
import ParakeetStreamingASR // Streaming dictation with partials + EOU
import NemotronStreamingASR // Multilingual streaming ASR with native punctuation (0.6B, 40 langs)
import OmnilingualASR       // 1,672 languages (CoreML + MLX)
import CohereTranscribeASR  // Cohere Transcribe 2B (MLX, 14 languages)
import VoxtralASR           // Voxtral Mini 3B (MLX, 8 languages)
import Qwen3TTS             // Text-to-speech
import CosyVoiceTTS         // Text-to-speech with voice cloning
import VoxCPM2TTS           // 48 kHz TTS with voice cloning + voice design (2B)
import IndexTTS2TTS         // Native MLX voice cloning from reference audio
import F5TTS                // Zero-shot voice cloning (DiT flow matching + Vocos)
import HiggsTTS             // Conversational TTS + cloning (Qwen3 backbone, control tags)
import FireRedTTS3TTS       // FireRedTTS3-Instruct (experimental, MLX FP32)
import MossVoiceGeneratorTTS // MOSS-VoiceGenerator (experimental, MLX BF16 + FP32 codec)
import CSM                  // Conversational Speech Model — text→audio + voice cloning (Sesame CSM-1B, MLX)
import KokoroTTS            // Text-to-speech (iOS-ready)
import VibeVoiceTTS         // Long-form / multi-speaker TTS (EN/ZH)
import MagpieTTS            // Multilingual TTS (NVIDIA Magpie 357M, MLX, 9 langs)
import MagpieTTSCoreML      // Magpie CoreML backend (hybrid CoreML + MLX, 8 langs)
import FishAudioTTS         // Experimental Fish Audio S2 Pro runtime with voice cloning
import IndicMioTTS          // Hindi/Indic TTS with emotion markers
import Qwen3Chat            // On-device LLM chat
import FunctionGemma    // On-device tool-call LLM
import Clef                // Local decisions from text
import MADLADTranslation    // Many-to-many translation across 400+ languages
import HibikiTranslate      // Streaming speech-to-speech translation (FR/ES/PT/DE → EN)
import PersonaPlex          // Full-duplex speech-to-speech
import SpeechVAD            // VAD + speaker diarization + embeddings
import SpeechWakeWord       // Wake-word / keyword spotting
import SpeechEnhancement    // Noise suppression
import SpeechRestoration    // Speech restoration — denoise + dereverb (Sidon, CoreML, 48 kHz)
import SourceSeparation     // Music source separation (Open-Unmix, 4 stems)
import StableAudio3MusicGen // Text-to-audio/music generation (Stable Audio 3)
import SpeechUI             // SwiftUI components for streaming transcripts
import AudioCommon          // Shared protocols and utilities

Requirements

The macOS 15 / iOS 18 minimum comes from MLState — Apple's persistent ANE state API used by the CoreML pipelines (Qwen3-ASR, Qwen3-Chat, Qwen3-TTS) to keep KV caches resident on the Neural Engine across token steps.

Build from source

git clone https://github.com/soniqo/speech-swift
cd speech-swift
make build

make build compiles the Swift package and the MLX Metal shader library. The Metal library is required for GPU inference — without it you'll see Failed to load the default metallib at runtime. make debug for debug builds, make test for the test suite.

Full build and install guide →

Demo apps

Run the VoiceChat reminders demo after a release build:

./.build/release/speech voice-chat \
  --model /path/to/voicechat-mlx-int5 \
  --mcp-config Examples/VoiceChatMCP/apple-reminders.json

Add --debug-timeline for phrase, generated-pronunciation-end, and model-decoded tool lifecycle timestamps. It can reveal tool arguments, so keep it out of shared logs. See each app's README or the linked VoiceChat guide for build and runtime details.

Code examples

The snippets below show the minimal path for each domain. Every section links to a full guide on soniqo.audio with configuration options, multiple backends, streaming patterns, and CLI recipes.

Speech-to-Text — full guide →

import Qwen3ASR

let model = try await Qwen3ASRModel.fromPretrained()
let text = model.transcribe(audio: audioSamples, sampleRate: 16000)

Alternative backends: WhisperASR (Whisper Large-v3 Turbo, native CoreML), Parakeet TDT (CoreML, 32× realtime), Omnilingual ASR (1,672 languages, CoreML or MLX), Streaming dictation (live partials).

Forced Alignment — full guide →

import Qwen3ASR

let aligner = try await Qwen3ForcedAligner.fromPretrained()
let aligned = aligner.align(
    audio: audioSamples,
    text: "Can you guarantee that the replacement part will be shipped tomorrow?",
    sampleRate: 24000
)
for word in aligned {
    print("[\(word.startTime)s - \(word.endTime)s] \(word.text)")
}

Text-to-Speech — full guide →

import Qwen3TTS
import AudioCommon

let model = try await Qwen3TTSModel.fromPretrained()
let audio = model.synthesize(text: "Hello world", language: "english")
try WAVWriter.write(samples: audio, sampleRate: 24000, to: outputURL)

Alternative TTS engines: CosyVoice3 (streaming + voice cloning + emotion tags), Kokoro-82M (iOS-ready, 54 voices), VibeVoice (long-form podcast / multi-speaker, EN/ZH), Fish Audio S2 Pro (experimental zero-shot cloning + bracket style markers), Voice cloning.

Speech-to-Speech — full guide →

import PersonaPlex

let model = try await PersonaPlexModel.fromPretrained()
let responseAudio = model.respond(userAudio: userSamples)
// 24 kHz mono Float32 output ready for playback

LLM Chat — full guide →

import Qwen3Chat
import FunctionGemma

let chat = try await Qwen35MLXChat.fromPretrained()
chat.chat(messages: [(.user, "Explain MLX in one sentence")]) { token, isFinal in
    print(token, terminator: "")
}

Local decisions with Clef-flash

Give Clef some text and a list of answers. It scores each answer so your app can choose the next step. Supports choices, true/false questions, and ordered scores. Add the Clef SPM product to use the library.

import Clef

let model = try await Clef.fromPretrained()
let result = try model.decide(
    state: "Please turn the kitchen lights on.",
    questions: [
        ClefQuestion(id: "action", instructions: "Which action was requested?",
            kind: .choice(["on": "Turn lights on", "off": "Turn lights off", "other": "Other request"]))
    ])
print(result.decisions[0].selectedOption)
print(result.decisions[0].probabilities)

Requires Apple Silicon and macOS 15+. The first load downloads about 5.3 GB; runtime memory is additional. Keep one instance loaded and use it serially.

Model · Usage · Measured latency and memory

Translation — full guide →

import MADLADTranslation

let translator = try await MADLADTranslator.fromPretrained()
let es = try translator.translate("Hello, how are you?", to: "es")
// → "Hola, ¿cómo estás?"

Speech Translation — full guide →

import HibikiTranslate
import AudioCommon

let model = try await HibikiTranslateModel.fromPretrained()
let pcm = try AudioFileLoader.load(url: input, targetSampleRate: 24000)
let (englishAudio, textTokens) = model.translate(
    sourceAudio: pcm, sourceLanguage: .fr
)
// Hibiki Zero-3B — FR/ES/PT/DE → EN, on-device, streaming Mimi codec

Voice Activity Detection — full guide →

import SpeechVAD

let vad = try await SileroVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for s in segments { print("\(s.startTime)s → \(s.endTime)s") }

Speaker Diarization — full guide →

import SpeechVAD

let diarizer = try await DiarizationPipeline.fromPretrained()
let segments = diarizer.diarize(audio: samples, sampleRate: 16000)
for s in segments { print("Speaker \(s.speakerId): \(s.startTime)s - \(s.endTime)s") }

Speech Enhancement — full guide →

import SpeechEnhancement

let denoiser = try await SpeechEnhancer.fromPretrained()             // CoreML (Neural Engine)
// let denoiser = try await SpeechEnhancer.fromPretrained(engine: .mlx)  // MLX (GPU, fp32)
let clean = try denoiser.enhance(audio: noisySamples, sampleRate: 48000)

Acoustic Echo Cancellation — full guide →

import SpeechEnhancement

let aec = try await LocalVQEEchoCanceller.fromPretrained()
let cleanMicrophone = try aec.processFrame(
    microphone: microphoneFrame,
    reference: playbackReferenceFrame
)

Speech Restoration — full guide →

Joint denoise and dereverb with Sidon (w2v-BERT 2.0 predictor + DAC vocoder, Core ML). Unlike a generic noise suppressor, Sidon is trained to preserve speaker identity, so it is well suited to cleaning a noisy or reverberant voice-cloning reference before TTS. Input is 16 kHz; output is 48 kHz mono.

import SpeechRestoration

let restorer = try await SpeechRestorer.fromPretrained()          // .fp16 (default) or .int8
let clean = try restorer.restore(audio: noisySamples, sampleRate: 16000)  // → 48 kHz

From the CLI:

speech restore noisy.wav -o clean.wav            # denoise + dereverb, 48 kHz output
speech restore noisy.wav --variant int8          # smaller, lower peak RAM

# Clean a voice-cloning reference before TTS (opt-in; preserves speaker identity):
speech speak "Hello" --engine voxcpm2 --voice-sample ref.wav --clean-reference

Voice Pipeline (ASR → LLM → TTS) — full guide →

import SpeechCore

let pipeline = VoicePipeline(
    stt: parakeetASR,
    tts: qwen3TTS,
    vad: sileroVAD,
    config: .init(mode: .voicePipeline),
    onEvent: { event in print(event) }
)
pipeline.start()
pipeline.pushAudio(micSamples)

VoicePipeline is the real-time voice-agent state machine (powered by speech-core) with VAD-driven turn detection, interruption handling, and eager STT. It connects any SpeechRecognitionModel + SpeechGenerationModel + StreamingVADProvider.

HTTP API server

speech-server --port 8080

Exposes every model via HTTP REST + WebSocket endpoints, including OpenAI-compatible APIs: a Realtime WebSocket at /v1/realtime and a transcription REST endpoint at /v1/audio/transcriptions. Realtime sessions use explicit input_audio_buffer.commit by default; send session.update with turn_detection.type set to server_vad for Silero-based automatic speech start/end detection, prefix padding, and bounded turn length. See Sources/AudioServer/ and Shared Protocols.

Papers

Architecture

speech-swift is split into one SPM target per model so consumers only pay for what they import. Shared infrastructure lives in AudioCommon (protocols, audio I/O, HuggingFace downloader, SentencePieceModel) and MLXCommon (weight loading, QuantizedLinear helpers, SDPA multi-head attention helper).

Full architecture diagram with backends, memory tables, and module map → soniqo.audio/architecture · API reference → soniqo.audio/api · Benchmarks → soniqo.audio/benchmarks

Local docs (repo):

Cache configuration

Model weights download from HuggingFace on first use and cache to ~/Library/Caches/qwen3-speech/. Override with QWEN3_CACHE_DIR (CLI) or cacheDir: (Swift API). All fromPretrained() entry points also accept offlineMode: true to skip network when weights are already cached.

Users in mainland China (or anywhere huggingface.co is slow/blocked) can fetch from a mirror by setting HF_ENDPOINT, e.g. export HF_ENDPOINT=https://hf-mirror.com.

See docs/inference/cache-and-offline.md for full details including sandboxed iOS container paths.

MLX Metal library

If you see Failed to load the default metallib at runtime, the Metal shader library is missing. Run make build or ./scripts/build_mlx_metallib.sh release after a manual swift build. If the Metal Toolchain is missing, install it first:

xcodebuild -downloadComponent MetalToolchain

Testing

make test                            # full suite (unit + E2E with model downloads)
swift test --skip E2E                # unit only (CI-safe, no downloads)
swift test --filter Qwen3ASRTests    # specific module

E2E test classes use the E2E prefix so CI can filter them out with --skip E2E. See CLAUDE.md for the full testing convention.

Contributing

PRs welcome — bug fixes, new model integrations, documentation. Fork, create a feature branch, make build && make test, open a PR against main.

License

Apache 2.0

speech-swift (soniqo/speech-swift)

正在搜索数据库…

这是什么?Speech Swift 是一组面向 Apple Silicon 的本地设备端 AI 语音模型合集,涵盖语音识别(ASR)、语音合成、语音到语音转换、语音活动检测和说话人分离。它以 Swift 包的形式构建,由 MLX 和 CoreML 驱动。
这是一个库,还是一堆独立的包?它是一个由数十个独立 Swift Package Manager 库组成的合集(每个模型一个,例如 Qwen3ASR、KokoroTTS、VoxCPM2TTS),外加共享的 SpeechCore、SpeechUI 模块,因此应用只需链接它实际使用的模型。
我该如何把它添加到我的 Xcode 项目中?在 Package.swift 中把该仓库添加为 Swift Package 依赖,然后只导入你需要的具体模型产品(例如 WhisperASR、Qwen3TTS);转写一段音频缓冲区大约只需 3 行代码。
我需要 Apple Silicon 吗?能在 Intel 芯片的 Mac 上运行吗?必须使用 Apple Silicon。这些模型依赖 MLX 以及 CoreML 和神经网络引擎加速,而且该项目专门面向 Mac 和 iOS。
这些模型都可以免费用于商业用途吗?大多数模型采用 Apache-2.0 等宽松许可证,但其中少数(如 F5-TTS 和 Higgs TTS 3)仅限非商业或研究用途。因此在将任何模型集成到产品中发布之前,都需要核实该模型的许可证。
它能在 iOS 上运行,还是只能在 Mac 上用?两者都支持。该库同时面向 macOS 和 iOS,其中几个模型还专门在 iPhone 16 Pro 和 M5 Pro 硬件上做过基准测试。
我使用这些模型时,我的音频数据会传到哪里?不会离开设备。所有模型都在 Apple Silicon 上本地运行,没有云端调用,也不涉及 API 密钥。
开箱即用包含什么?包含数十个模型,涵盖语音识别、语音合成、语音到语音转换、翻译、语音活动检测、说话人分离、音源分离和音乐生成。每个模型都作为独立的 SPM 产品提供,因此应用只需打包它实际使用的部分。
MLX 和 CoreML 后端在这个库里有什么区别?MLX 模型通过 Apple 的 MLX 框架在 GPU 上运行,CoreML 模型则在神经网络引擎(ANE)上运行。项目博客明确比较了不同工作负载下该选择哪种后端。
它是否包含现成的界面组件?SpeechUI 提供 TranscriptionView 和 TranscriptionStore,用于构建听写界面,可显示最终结果和实时中间结果。不过音频采集和播放仍需要使用 AVFoundation。
除了转写和语音合成,它还能做更多吗?可以。它还支持语音到语音翻译、降噪、回声消除、语音克隆、唤醒词检测以及文本生成音乐。
作者
soniqo
大小
16.6 MB
星标
1.3K
时长
9个月
提交 · 90 天