DramaBox README

The project's README from github.com/resemble-ai/DramaBox. As one markdown file: README.md.

DramaBox

DramaBox — Expressive TTS with Voice Cloning

Alt Text Discord

Built on LTX-2 by Lightricks. DramaBox is Resemble AI's expressive TTS, trained on top of the LTX-2.3 audio branch under the LTX-2 Community License. Huge thanks to the Lightricks team for open-sourcing the base.

Made with ♥️ by resemble-logo-horizontal

Prompt-driven TTS with voice cloning. The prompt itself controls speaker identity, emotion, delivery style, laughs, sighs, pauses and transitions; an optional 10-second voice reference clones the target timbre. DramaBox is an IC-LoRA fine-tune of the LTX-2.3 3.3B audio-only model.
🤗 Model ResembleAI/Dramabox
🎭 Demo Space ResembleAI/Dramabox (ZeroGPU)
🏗️ Base model Lightricks/LTX-2.3
📜 License LTX-2 Community License — see LICENSE

Models

Auto-downloaded from the HF model repo on first run.

File Size Description
dramabox-dit-v1.safetensors 6.6 GB DiT transformer (LoRA already merged into base)
dramabox-audio-components.safetensors 1.9 GB Audio embeddings connector + audio text projection + audio VAE + vocoder
unsloth/gemma-3-12b-it-bnb-4bit ~8 GB Text encoder

VRAM: ~24 GB peak · Speed: ~2.5 s / generation (warm server, H100)

Quick Start

from src.inference_server import TTSServer

server = TTSServer(device="cuda")

server.generate_to_file(
    prompt='A woman speaks warmly, "Hello, how are you today?" She laughs, "Hahaha, it is so good to see you!"',
    output="output.wav",
    voice_ref="reference.wav",   # optional, 10+ seconds
)

CLI

python src/inference.py \
  --voice-sample reference.wav \
  --prompt 'A woman speaks warmly, "Hello, how are you today?"' \
  --output output.wav \
  --cfg-scale 2.5 --stg-scale 1.5

Gradio app

CUDA_VISIBLE_DEVICES=4 python app.py

Inference Settings

Parameter Default Notes
cfg-scale 2.5 Lower = more natural, higher = more text-faithful
stg-scale 1.5 Skip-token guidance
rescale 0 No rescaling
modality 1 No modality guidance
duration-multiplier 1.1 10% breathing room on auto-estimated length
steps 30 Euler flow matching

Prompt Writing Guide

Structure: <speaker description>, "<dialogue>" <action direction> "<more dialogue>"

Inside quotes (model produces actual sounds):

Outside quotes (stage directions):

Avoid inside quotes (model speaks them literally): Ahem, Pfft, Sigh, Gasp, Cough.

Tips

Watermarking

Every audio output from inference.py and inference_server.TTSServer.generate_to_file is automatically watermarked with Resemble Perth — an imperceptible neural watermark that survives MP3 compression, audio editing, and common manipulations while maintaining nearly 100% detection accuracy.

import perth, librosa
wav, sr = librosa.load("output.wav", sr=None, mono=True)
detector = perth.PerthImplicitWatermarker()
print(detector.get_watermark(wav, sample_rate=sr))   # confidence ≈ 1.0

Pass --no-watermark to inference.py (or watermark=False to generate_to_file) to disable for debugging.

Voice reference denoising (RE-USE)

Opt-in. RE-USE deps (mamba-ssm, causal-conv1d, librosa, resampy) are not installed by default — mamba-ssm / causal-conv1d have no pre-built wheels on macOS / Windows and require matching CUDA + nvcc on Linux, which breaks many fresh installs. The base requirements.txt skips them; install only when you want voice-reference denoising:

pip install -r requirements-reuse.txt

If the deps are missing at runtime, the server logs a one-line "denoise disabled — pip install -r requirements-reuse.txt to enable" warning and continues generation without denoising the reference.

When installed, the voice reference is denoised with nvidia/RE-USE before VAE conditioning. On (defaults to True):

server.generate_to_file(
    prompt='A woman speaks warmly, "Hello."',
    output="out.wav",
    voice_ref="ref.wav",
    denoise_ref=True,
)

Setup is automatic — code (.py / .yaml, 150 KB) is snapshot-downloaded into `/.cache/dramabox/ on first call; weights (~38 MB) come from SEMamba.from_pretrainedthrough the standard HF cache. Pass$REUSE_DIRor vendor atthird_party/RE-USE/` to skip the download.

If the mamba-ssm / causal-conv1d wheels fail to build on a supported Linux box, the wrapper falls back to a pure-PyTorch path (5-10× slower, same output). RE-USE is NSCLv1 (non-commercial) — set denoise_ref=False to skip.

Long-form generation (text chunking)

The base LTX-2.3 audio DiT was trained on clips ≤ ~20 s. Generations up to ~45 s remain usable thanks to the silence-prior patch in inference_server.py, but quality drifts past that. For arbitrarily long prompts, TTSServer now chunks automatically:

server.generate_to_file(
    prompt=very_long_scene,        # 2 minutes worth of dialogue is fine
    output="long.wav",
    voice_ref="ref.wav",           # reused across every chunk → speaker stays coherent
)

The chunker (src/text_chunker.py:chunk_prompt_for_duration) splits at sentence / quote-group boundaries, preserves the speaker-description prefix on every chunk, and targets ~37 s per chunk (≤45 s hard cap). TTSServer.generate_long then concatenates the per-chunk waveforms with a 50 ms equal-power crossfade so joins are inaudible. Set max_chunk_duration= / target_chunk_duration= on generate_to_file to tune.

Training a LoRA on top of DramaBox

You can fine-tune your own LoRA using DramaBox itself as the base — no need to start from raw LTX-2.3. Useful for adding a specific speaker, language flavour, or style on top of the existing expressive prior.

1. Prepare your index file

The preprocessor accepts four formats. The text field is the target transcript; if you want to attach a scene-style prompt (the part the model conditions on at inference time), prepend it to the transcript in the same format the model was trained on:

A woman speaks warmly, "<your transcript here>"

Both forms are supported — with or without the prompt wrapper. Without the wrapper the model treats the entry as plain text-to-speech.

Format A — manifest (JSONL) — recommended for new datasets:

{"audio_filepath": "wavs/spk01_001.wav", "text": "A woman speaks warmly, \"Hello, how are you today?\""}
{"audio_filepath": "wavs/spk01_002.wav", "text": "Hello, how are you today?"}
{"audio_filepath": "wavs/spk02_001.flac", "text": "An exhausted father sighs, \"Sweetie, daddy is asking very nicely.\"", "duration": 4.7}

Fields: audio_filepath (or audio_path) is required, text (or transcript) is required, duration is optional.

Format B — tsv — simplest, one line per sample:

wavs/spk01_001.wav	A woman speaks warmly, "Hello, how are you today?"
wavs/spk01_002.wav	Hello, how are you today?

Format C — gemini_synthetic — ~-separated, used for prompted synthetic data:

id~speaker~lang~sr~samples~dur~phonemes~text
spk01_001~spk01~en~24000~93000~3.875~_~A woman speaks warmly, "Hello, how are you today?"

Format D — libriheavy — ~-separated, for unprompted text-only data:

id~speaker~lang~samples~dur_ms~phonemes~text
spk01_001~spk01~en~93000~3875~_~Hello, how are you today?

2. Preprocess

python src/preprocess.py \
  --dataset-type manifest \
  --index your_data.jsonl \
  --audio-dir /path/to/wavs \
  --output-dir /path/to/preprocessed/ \
  --checkpoint /path/to/dramabox-audio-components.safetensors \
  --gemma-root /path/to/gemma-3-12b-it-bnb-4bit/ \
  --max-duration 20.0 --min-duration 2.0

Output layout (training-ready .pt files):

preprocessed/
├── audio_latents/sample_*.pt     # Audio VAE-encoded latents
├── conditions/sample_*.pt        # Gemma text embeddings
└── latents/sample_*.pt           # Dummy video latents (placeholder)

3. Train

Copy configs/training_args.example.yaml, point data_dir / speaker_index at your preprocessed output, set checkpoint + full_checkpoint to the DramaBox files, then launch with HuggingFace accelerate. Any flag passed on the CLI overrides the YAML.

accelerate launch src/train.py \
  --config configs/training_args.example.yaml

The trainer attaches a fresh LoRA to the audio branch on top of the DramaBox checkpoint. LoRA targets: audio_attn1.{to_q,to_k,to_v,to_out.0} + audio_ff.{net.0.proj,net.2} × 48 transformer blocks (288 LoRA pairs total). Default rank 128 / alpha 128 / dropout 0.1, cosine LR schedule from 1e-4 with 500-step warmup over 10k steps.

To monitor training, set val_config: configs/val_config.example.yaml in your training YAML — src/validate.py is then spawned at every save step to generate one wav per speaker entry, so you can A/B listen during the run.

Inference with your trained LoRA

python src/inference.py \
  --lora /path/to/your/lora_step_5000.safetensors \
  --voice-sample reference.wav \
  --prompt 'A woman speaks warmly, "..."' \
  --output output.wav

Always load the LoRA at inference rather than pre-merging it — pre-merged checkpoints have produced degraded output in our runs.

Language

English.

License & acknowledgement

DramaBox is a Resemble AI fine-tune of LTX-2. Distributed under the LTX-2 Community License Agreement — see LICENSE. Thanks again to Lightricks for releasing the base model.

DramaBox (resemble-ai/DramaBox)

Searching Database…

507 ·GitHub
DramaBox
Что это?DramaBox — выразительная модель синтеза речи с клонированием голоса от Resemble AI, дообученная поверх аудиомодели LTX-2.3 от Lightricks. Сам промпт управляет личностью говорящего, эмоцией, манерой подачи, смехом, вздохами, паузами и переходами, а опциональная 10-секундная голосовая референс-запись клонирует тембр целевого голоса.
Это хостинговый сервис или то, что я запускаю сам?Это модель для самостоятельного размещения: вы запускаете её через warm-сервер, CLI или Gradio-приложение. Код и веса автоматически скачиваются с Hugging Face в локальный кэш при первом запуске.
Как её запустить?Используйте раздел Quick Start: запустите warm-сервер (рекомендуется), CLI или Gradio-приложение. Веса модели (DiT-трансформер, аудиокомпоненты, текстовый энкодер) автоматически скачиваются из репозитория Hugging Face при первом запуске.
Какое железо нужно для запуска?Рассчитывайте примерно на 24 ГБ пиковой VRAM и около 2,5 секунды на генерацию на H100. Опциональное шумоподавление голосового референса (RE-USE) тянет дополнительные зависимости, которые по умолчанию не установлены.
Есть ли ограничение на длину генерируемой речи?Базовая аудиомодель обучалась на клипах примерно по 20 секунд и короче; вывод остаётся пригодным до ~45 секунд, но дальше качество уплывает. Для более длинных промптов сервер автоматически режет текст на куски и склеивает их кроссфейдом.
Работает ли шумоподавление голосового референса и на macOS и Windows?Не из коробки — зависимости шумоподавления (mamba-ssm, causal-conv1d) не имеют готовых wheel-пакетов на macOS и Windows и требуют подходящую CUDA на Linux, поэтому по умолчанию они пропускаются; без них на поддерживаемых Linux-машинах используется более медленный чистый путь на PyTorch.
Как на самом деле выглядит промпт для DramaBox?Промпт сочетает описание говорящего с диалогом в кавычках и сценическими ремарками: звуки вроде смеха или вздохов пишутся внутри кавычек, чтобы модель озвучивала их буквально, а действия вроде «Он прочищает горло» идут вне кавычек как ремарки.
Сгенерированные аудиофайлы с водяными знаками?Да, весь вывод скрипта инференса или сервера помечается водяным знаком Resemble Perth — незаметным нейронным водяным знаком, который переживает MP3-сжатие и обычное редактирование; для отладки его можно отключить флагом.
Можно ли дообучить свой голос или стиль поверх DramaBox?Да, можно обучить новую LoRA поверх самого DramaBox, используя приложенные скрипты предобработки для нескольких форматов датасетов и скрипт обучения Hugging Face Accelerate, не начиная с сырой базы LTX-2.3.
Какой язык поддерживает модель?Только английский, согласно README.
What are similar projects?bark, VoxCPM, OmniVoice, csm, dia, FireRedAudio, MOSS-TTS, WanGP
Which topics is it in?text-to-audio, audio-generation, speech-synthesis, voice-cloning, gradio, lora, tts, text-to-speech, fine-tuning, transformer, audio, nvidia
Creator
resemble-ai
Size
6.7 MB
Stars
507
Updated
CommitMay 23
Age
6mo
Commits · 90 Days