subvox README

项目的 README,来自 github.com/joeblau/subvox。合并为一个 markdown 文件:README.md。

Subvox

A macOS prototype that tests whether AirPods Pro head-motion data (CMHeadphoneMotionManager) carries enough information to turn silently mouthed words into text.

It doesn't use a language model. It's a small-vocabulary template recognizer plus a statistical test that tells you whether the signal is real.

Build & run

brew install xcodegen   # once
xcodegen generate
open Subvox.xcodeproj   # ⌘R

Or from the command line:

xcodebuild -project Subvox.xcodeproj -scheme Subvox -derivedDataPath build -allowProvisioningUpdates build
open build/Build/Products/Debug/Subvox.app
xcodebuild -project Subvox.xcodeproj -scheme Subvox -derivedDataPath build test

Requires macOS 14+, AirPods Pro or another headphone that supports head tracking, connected as the Mac's audio output. On first launch, grant Motion. The mic and speech prompts only appear when you start the mic.

Screens

Screen What it does
Sensors Live plots of user acceleration, rotation rate and attitude, the measured sample rate, a mic input picker and level meter, and a recorder for experiment sessions.
Train Pick a word, hold Space, mouth it, release. Each take is stored as a template. Evaluate runs leave-one-out classification and a 500× label-permutation test, then shows accuracy against chance, a p-value and a confusion matrix. The feature checkboxes (acceleration / rotation / attitude) let you test which channels carry the signal.
Decode The motion-to-text screen. Hold Space, or use auto-detect after calibrating the noise floor. Each segment is DTW-matched against your templates and added to the transcript. A live mic transcript runs alongside it for comparison.

How decoding works

  1. Segmentation (CaptureController). Space-hold, or an energy trigger above a calibrated noise floor with pre-roll and hangover.
  2. Features (MotionFeatures). Per frame: user acceleration (3), rotation rate (3), and attitude change since the start of the segment (3). Each channel is mean-centered, and quiet lead-in and lead-out frames are trimmed.
  3. Matching (DTW, TemplateClassifier). Each channel is z-scaled across the training set. Matching is 1-nearest-neighbour per word, using dynamic time warping with a Sakoe–Chiba band. A segment is rejected if the best distance is above max distance or the margin to the runner-up is below min confidence.
  4. Is it real? (Evaluation). The leave-one-out accuracy is compared with the same procedure run on shuffled labels. If p < 0.05, the motion stream distinguishes your words better than chance.

Experiment protocol

Use Record Session in the toolbar or on the Sensors screen. Recordings go to ~/Documents/Subvox/Sessions/<timestamp>-<condition>/:

Suggested runs: the same 5-word script under Normal speech, Whisper and Silent mouthing, 10 reps each. Then, in Train, compare the Evaluate results across feature sets.

The training set lives at ~/Library/Application Support/Subvox/dataset.json.

Known limits

subvox (joeblau/subvox)

正在搜索数据库…

839 ·GitHub
这是什么?Subvox 是一个 macOS 研究原型,用于验证从 AirPods Pro(通过 CMHeadphoneMotionManager)采集的头部运动数据是否包含足够的信号,从而识别无声默念的词语并将其转换为文字。
这是成品还是实验项目?这是一个实验性的研究工具,而不是成品:它结合了小词汇量的模板识别器和统计显著性检验,完全不涉及语言模型。
我该如何构建和运行它?可以通过 Xcode 或命令行在 macOS 上构建并运行。首次启动时需要授予“运动”权限;只有在真正开启麦克风后,麦克风和语音相关的提示才会出现。
使用它需要什么硬件?需要一台运行 macOS 14 或更高版本的 Mac,以及支持头部追踪的 AirPods Pro 或其他耳机,并将其连接为 Mac 的音频输出设备。
无声语音检测的可靠性如何?耳机运动传感器的采样率较低(AirPods 约为 25Hz),只能捕捉下颌和头部的运动,无法捕捉真实的声带振动,因此识别模板因用户和会话而异,如果重新佩戴 AirPods,模板可能会出现偏移。
它是否使用 AI 或语言模型来识别词语?不是。Subvox 依靠基于动态时间规整(DTW)的小词汇量模板识别器,不使用语言模型,甚至还运行统计置换检验,以验证识别结果是否真实有效。
一次训练或解码会话会生成哪些数据?每次会话会保存 motion.csv(每个运动采样点)、audio.caf(如果运行时开启了麦克风则包含麦克风音频)、events.csv(每个片段及其标签或预测结果)和 meta.json;训练好的词语模板保存在 ~/Library/Application Support/Subvox/dataset.json 中。
如何判断检测到的信号是否具有统计意义?在“训练”界面中,留一法分类准确率会与在 500 次打乱标签的置换结果上运行相同流程所得的结果进行比较,从而给出准确率与随机水平的对比、p 值和混淆矩阵。
作者
joeblau
大小
<0.1 MB
星标
839
更新
提交Oct 31
时长
18天
提交 · 90 天