DramaBox — Expressive TTS with Voice Cloning
Built on LTX-2 by Lightricks.
DramaBox is Resemble AI's expressive TTS, trained on top of the LTX-2.3 audio branch under the LTX-2 Community License. Huge thanks to the Lightricks team for open-sourcing the base.
Made with ♥️ by 
Prompt-driven TTS with voice cloning. The prompt itself controls speaker identity, emotion, delivery style, laughs, sighs, pauses and transitions; an optional 10-second voice reference clones the target timbre. DramaBox is an IC-LoRA fine-tune of the LTX-2.3 3.3B audio-only model.
Models
Auto-downloaded from the HF model repo on first run.
| File |
Size |
Description |
dramabox-dit-v1.safetensors |
6.6 GB |
DiT transformer (LoRA already merged into base) |
dramabox-audio-components.safetensors |
1.9 GB |
Audio embeddings connector + audio text projection + audio VAE + vocoder |
unsloth/gemma-3-12b-it-bnb-4bit |
~8 GB |
Text encoder |
VRAM: ~24 GB peak · Speed: ~2.5 s / generation (warm server, H100)
Quick Start
Warm server (recommended)
from src.inference_server import TTSServer
server = TTSServer(device="cuda")
server.generate_to_file(
prompt='A woman speaks warmly, "Hello, how are you today?" She laughs, "Hahaha, it is so good to see you!"',
output="output.wav",
voice_ref="reference.wav", # optional, 10+ seconds
)
CLI
python src/inference.py \
--voice-sample reference.wav \
--prompt 'A woman speaks warmly, "Hello, how are you today?"' \
--output output.wav \
--cfg-scale 2.5 --stg-scale 1.5
Gradio app
CUDA_VISIBLE_DEVICES=4 python app.py
Inference Settings
| Parameter |
Default |
Notes |
cfg-scale |
2.5 |
Lower = more natural, higher = more text-faithful |
stg-scale |
1.5 |
Skip-token guidance |
rescale |
0 |
No rescaling |
modality |
1 |
No modality guidance |
duration-multiplier |
1.1 |
10% breathing room on auto-estimated length |
steps |
30 |
Euler flow matching |
Prompt Writing Guide
Structure: <speaker description>, "<dialogue>" <action direction> "<more dialogue>"
Inside quotes (model produces actual sounds):
- Laughs:
"Hahaha" "Hehehe" (always one word, never separated)
- Sounds:
"Mmmmm" "Ugh" "Argh" "Ahhh" "Hmm"
Outside quotes (stage directions):
She sighs deeply. · He gulps nervously. · A long pause.Her voice cracks. · He clears his throat. · She scoffs.
Avoid inside quotes (model speaks them literally): Ahem, Pfft, Sigh, Gasp, Cough.
Tips
- Match gender/age in the speaker description to the voice reference
- Break long dialogue into segments with action directions in between
- End the prompt at the last closing quote mark (no trailing description)
Watermarking
Every audio output from inference.py and inference_server.TTSServer.generate_to_file is automatically watermarked with Resemble Perth — an imperceptible neural watermark that survives MP3 compression, audio editing, and common manipulations while maintaining nearly 100% detection accuracy.
import perth, librosa
wav, sr = librosa.load("output.wav", sr=None, mono=True)
detector = perth.PerthImplicitWatermarker()
print(detector.get_watermark(wav, sample_rate=sr)) # confidence ≈ 1.0
Pass --no-watermark to inference.py (or watermark=False to generate_to_file) to disable for debugging.
Voice reference denoising (RE-USE)
Opt-in. RE-USE deps (mamba-ssm, causal-conv1d, librosa, resampy)
are not installed by default — mamba-ssm / causal-conv1d have no
pre-built wheels on macOS / Windows and require matching CUDA + nvcc on
Linux, which breaks many fresh installs. The base requirements.txt skips
them; install only when you want voice-reference denoising:
pip install -r requirements-reuse.txt
If the deps are missing at runtime, the server logs a one-line "denoise
disabled — pip install -r requirements-reuse.txt to enable" warning and
continues generation without denoising the reference.
When installed, the voice reference is denoised with nvidia/RE-USE before VAE conditioning. On (defaults to True):
server.generate_to_file(
prompt='A woman speaks warmly, "Hello."',
output="out.wav",
voice_ref="ref.wav",
denoise_ref=True,
)
Setup is automatic — code (.py / .yaml, 150 KB) is snapshot-downloaded into `/.cache/dramabox/ on first call; weights (~38 MB) come from SEMamba.from_pretrainedthrough the standard HF cache. Pass$REUSE_DIRor vendor atthird_party/RE-USE/` to skip the download.
If the mamba-ssm / causal-conv1d wheels fail to build on a supported Linux box, the wrapper falls back to a pure-PyTorch path (5-10× slower, same output). RE-USE is NSCLv1 (non-commercial) — set denoise_ref=False to skip.
Long-form generation (text chunking)
The base LTX-2.3 audio DiT was trained on clips ≤ ~20 s. Generations up to ~45 s remain usable thanks to the silence-prior patch in inference_server.py, but quality drifts past that. For arbitrarily long prompts, TTSServer now chunks automatically:
server.generate_to_file(
prompt=very_long_scene, # 2 minutes worth of dialogue is fine
output="long.wav",
voice_ref="ref.wav", # reused across every chunk → speaker stays coherent
)
The chunker (src/text_chunker.py:chunk_prompt_for_duration) splits at sentence / quote-group boundaries, preserves the speaker-description prefix on every chunk, and targets ~37 s per chunk (≤45 s hard cap). TTSServer.generate_long then concatenates the per-chunk waveforms with a 50 ms equal-power crossfade so joins are inaudible. Set max_chunk_duration= / target_chunk_duration= on generate_to_file to tune.
Training a LoRA on top of DramaBox
You can fine-tune your own LoRA using DramaBox itself as the base — no need to start from raw LTX-2.3. Useful for adding a specific speaker, language flavour, or style on top of the existing expressive prior.
1. Prepare your index file
The preprocessor accepts four formats. The text field is the target transcript; if you want to attach a scene-style prompt (the part the model conditions on at inference time), prepend it to the transcript in the same format the model was trained on:
A woman speaks warmly, "<your transcript here>"
Both forms are supported — with or without the prompt wrapper. Without the wrapper the model treats the entry as plain text-to-speech.
Format A — manifest (JSONL) — recommended for new datasets:
{"audio_filepath": "wavs/spk01_001.wav", "text": "A woman speaks warmly, \"Hello, how are you today?\""}
{"audio_filepath": "wavs/spk01_002.wav", "text": "Hello, how are you today?"}
{"audio_filepath": "wavs/spk02_001.flac", "text": "An exhausted father sighs, \"Sweetie, daddy is asking very nicely.\"", "duration": 4.7}
Fields: audio_filepath (or audio_path) is required, text (or transcript) is required, duration is optional.
Format B — tsv — simplest, one line per sample:
wavs/spk01_001.wav A woman speaks warmly, "Hello, how are you today?"
wavs/spk01_002.wav Hello, how are you today?
Format C — gemini_synthetic — ~-separated, used for prompted synthetic data:
id~speaker~lang~sr~samples~dur~phonemes~text
spk01_001~spk01~en~24000~93000~3.875~_~A woman speaks warmly, "Hello, how are you today?"
Format D — libriheavy — ~-separated, for unprompted text-only data:
id~speaker~lang~samples~dur_ms~phonemes~text
spk01_001~spk01~en~93000~3875~_~Hello, how are you today?
2. Preprocess
python src/preprocess.py \
--dataset-type manifest \
--index your_data.jsonl \
--audio-dir /path/to/wavs \
--output-dir /path/to/preprocessed/ \
--checkpoint /path/to/dramabox-audio-components.safetensors \
--gemma-root /path/to/gemma-3-12b-it-bnb-4bit/ \
--max-duration 20.0 --min-duration 2.0
Output layout (training-ready .pt files):
preprocessed/
├── audio_latents/sample_*.pt # Audio VAE-encoded latents
├── conditions/sample_*.pt # Gemma text embeddings
└── latents/sample_*.pt # Dummy video latents (placeholder)
3. Train
Copy configs/training_args.example.yaml, point data_dir / speaker_index at your preprocessed output, set checkpoint + full_checkpoint to the DramaBox files, then launch with HuggingFace accelerate. Any flag passed on the CLI overrides the YAML.
accelerate launch src/train.py \
--config configs/training_args.example.yaml
The trainer attaches a fresh LoRA to the audio branch on top of the DramaBox checkpoint. LoRA targets: audio_attn1.{to_q,to_k,to_v,to_out.0} + audio_ff.{net.0.proj,net.2} × 48 transformer blocks (288 LoRA pairs total). Default rank 128 / alpha 128 / dropout 0.1, cosine LR schedule from 1e-4 with 500-step warmup over 10k steps.
To monitor training, set val_config: configs/val_config.example.yaml in your training YAML — src/validate.py is then spawned at every save step to generate one wav per speaker entry, so you can A/B listen during the run.
Inference with your trained LoRA
python src/inference.py \
--lora /path/to/your/lora_step_5000.safetensors \
--voice-sample reference.wav \
--prompt 'A woman speaks warmly, "..."' \
--output output.wav
Always load the LoRA at inference rather than pre-merging it — pre-merged checkpoints have produced degraded output in our runs.
Language
English.
License & acknowledgement
DramaBox is a Resemble AI fine-tune of LTX-2. Distributed under the LTX-2 Community License Agreement — see LICENSE. Thanks again to Lightricks for releasing the base model.
DramaBox 是 Resemble AI 的富有表现力的文本转语音模型，支持声音克隆，在 Lightricks 的 LTX-2.3 音频模型基础上微调而来。提示词本身就控制说话人身份、情绪、演绎风格、笑声、叹气、停顿和过渡，还可以用可选的 10 秒声音参考来克隆目标音色。