DramaBox README

The project's README from github.com/resemble-ai/DramaBox. As one markdown file: README.md.

DramaBox

DramaBox — Expressive TTS with Voice Cloning

Alt Text Discord

Built on LTX-2 by Lightricks. DramaBox is Resemble AI's expressive TTS, trained on top of the LTX-2.3 audio branch under the LTX-2 Community License. Huge thanks to the Lightricks team for open-sourcing the base.

Made with ♥️ by resemble-logo-horizontal

Prompt-driven TTS with voice cloning. The prompt itself controls speaker identity, emotion, delivery style, laughs, sighs, pauses and transitions; an optional 10-second voice reference clones the target timbre. DramaBox is an IC-LoRA fine-tune of the LTX-2.3 3.3B audio-only model.
🤗 Model ResembleAI/Dramabox
🎭 Demo Space ResembleAI/Dramabox (ZeroGPU)
🏗️ Base model Lightricks/LTX-2.3
📜 License LTX-2 Community License — see LICENSE

Models

Auto-downloaded from the HF model repo on first run.

File Size Description
dramabox-dit-v1.safetensors 6.6 GB DiT transformer (LoRA already merged into base)
dramabox-audio-components.safetensors 1.9 GB Audio embeddings connector + audio text projection + audio VAE + vocoder
unsloth/gemma-3-12b-it-bnb-4bit ~8 GB Text encoder

VRAM: ~24 GB peak · Speed: ~2.5 s / generation (warm server, H100)

Quick Start

from src.inference_server import TTSServer

server = TTSServer(device="cuda")

server.generate_to_file(
    prompt='A woman speaks warmly, "Hello, how are you today?" She laughs, "Hahaha, it is so good to see you!"',
    output="output.wav",
    voice_ref="reference.wav",   # optional, 10+ seconds
)

CLI

python src/inference.py \
  --voice-sample reference.wav \
  --prompt 'A woman speaks warmly, "Hello, how are you today?"' \
  --output output.wav \
  --cfg-scale 2.5 --stg-scale 1.5

Gradio app

CUDA_VISIBLE_DEVICES=4 python app.py

Inference Settings

Parameter Default Notes
cfg-scale 2.5 Lower = more natural, higher = more text-faithful
stg-scale 1.5 Skip-token guidance
rescale 0 No rescaling
modality 1 No modality guidance
duration-multiplier 1.1 10% breathing room on auto-estimated length
steps 30 Euler flow matching

Prompt Writing Guide

Structure: <speaker description>, "<dialogue>" <action direction> "<more dialogue>"

Inside quotes (model produces actual sounds):

Outside quotes (stage directions):

Avoid inside quotes (model speaks them literally): Ahem, Pfft, Sigh, Gasp, Cough.

Tips

Watermarking

Every audio output from inference.py and inference_server.TTSServer.generate_to_file is automatically watermarked with Resemble Perth — an imperceptible neural watermark that survives MP3 compression, audio editing, and common manipulations while maintaining nearly 100% detection accuracy.

import perth, librosa
wav, sr = librosa.load("output.wav", sr=None, mono=True)
detector = perth.PerthImplicitWatermarker()
print(detector.get_watermark(wav, sample_rate=sr))   # confidence ≈ 1.0

Pass --no-watermark to inference.py (or watermark=False to generate_to_file) to disable for debugging.

Voice reference denoising (RE-USE)

Opt-in. RE-USE deps (mamba-ssm, causal-conv1d, librosa, resampy) are not installed by default — mamba-ssm / causal-conv1d have no pre-built wheels on macOS / Windows and require matching CUDA + nvcc on Linux, which breaks many fresh installs. The base requirements.txt skips them; install only when you want voice-reference denoising:

pip install -r requirements-reuse.txt

If the deps are missing at runtime, the server logs a one-line "denoise disabled — pip install -r requirements-reuse.txt to enable" warning and continues generation without denoising the reference.

When installed, the voice reference is denoised with nvidia/RE-USE before VAE conditioning. On (defaults to True):

server.generate_to_file(
    prompt='A woman speaks warmly, "Hello."',
    output="out.wav",
    voice_ref="ref.wav",
    denoise_ref=True,
)

Setup is automatic — code (.py / .yaml, 150 KB) is snapshot-downloaded into `/.cache/dramabox/ on first call; weights (~38 MB) come from SEMamba.from_pretrainedthrough the standard HF cache. Pass$REUSE_DIRor vendor atthird_party/RE-USE/` to skip the download.

If the mamba-ssm / causal-conv1d wheels fail to build on a supported Linux box, the wrapper falls back to a pure-PyTorch path (5-10× slower, same output). RE-USE is NSCLv1 (non-commercial) — set denoise_ref=False to skip.

Long-form generation (text chunking)

The base LTX-2.3 audio DiT was trained on clips ≤ ~20 s. Generations up to ~45 s remain usable thanks to the silence-prior patch in inference_server.py, but quality drifts past that. For arbitrarily long prompts, TTSServer now chunks automatically:

server.generate_to_file(
    prompt=very_long_scene,        # 2 minutes worth of dialogue is fine
    output="long.wav",
    voice_ref="ref.wav",           # reused across every chunk → speaker stays coherent
)

The chunker (src/text_chunker.py:chunk_prompt_for_duration) splits at sentence / quote-group boundaries, preserves the speaker-description prefix on every chunk, and targets ~37 s per chunk (≤45 s hard cap). TTSServer.generate_long then concatenates the per-chunk waveforms with a 50 ms equal-power crossfade so joins are inaudible. Set max_chunk_duration= / target_chunk_duration= on generate_to_file to tune.

Training a LoRA on top of DramaBox

You can fine-tune your own LoRA using DramaBox itself as the base — no need to start from raw LTX-2.3. Useful for adding a specific speaker, language flavour, or style on top of the existing expressive prior.

1. Prepare your index file

The preprocessor accepts four formats. The text field is the target transcript; if you want to attach a scene-style prompt (the part the model conditions on at inference time), prepend it to the transcript in the same format the model was trained on:

A woman speaks warmly, "<your transcript here>"

Both forms are supported — with or without the prompt wrapper. Without the wrapper the model treats the entry as plain text-to-speech.

Format A — manifest (JSONL) — recommended for new datasets:

{"audio_filepath": "wavs/spk01_001.wav", "text": "A woman speaks warmly, \"Hello, how are you today?\""}
{"audio_filepath": "wavs/spk01_002.wav", "text": "Hello, how are you today?"}
{"audio_filepath": "wavs/spk02_001.flac", "text": "An exhausted father sighs, \"Sweetie, daddy is asking very nicely.\"", "duration": 4.7}

Fields: audio_filepath (or audio_path) is required, text (or transcript) is required, duration is optional.

Format B — tsv — simplest, one line per sample:

wavs/spk01_001.wav	A woman speaks warmly, "Hello, how are you today?"
wavs/spk01_002.wav	Hello, how are you today?

Format C — gemini_synthetic — ~-separated, used for prompted synthetic data:

id~speaker~lang~sr~samples~dur~phonemes~text
spk01_001~spk01~en~24000~93000~3.875~_~A woman speaks warmly, "Hello, how are you today?"

Format D — libriheavy — ~-separated, for unprompted text-only data:

id~speaker~lang~samples~dur_ms~phonemes~text
spk01_001~spk01~en~93000~3875~_~Hello, how are you today?

2. Preprocess

python src/preprocess.py \
  --dataset-type manifest \
  --index your_data.jsonl \
  --audio-dir /path/to/wavs \
  --output-dir /path/to/preprocessed/ \
  --checkpoint /path/to/dramabox-audio-components.safetensors \
  --gemma-root /path/to/gemma-3-12b-it-bnb-4bit/ \
  --max-duration 20.0 --min-duration 2.0

Output layout (training-ready .pt files):

preprocessed/
├── audio_latents/sample_*.pt     # Audio VAE-encoded latents
├── conditions/sample_*.pt        # Gemma text embeddings
└── latents/sample_*.pt           # Dummy video latents (placeholder)

3. Train

Copy configs/training_args.example.yaml, point data_dir / speaker_index at your preprocessed output, set checkpoint + full_checkpoint to the DramaBox files, then launch with HuggingFace accelerate. Any flag passed on the CLI overrides the YAML.

accelerate launch src/train.py \
  --config configs/training_args.example.yaml

The trainer attaches a fresh LoRA to the audio branch on top of the DramaBox checkpoint. LoRA targets: audio_attn1.{to_q,to_k,to_v,to_out.0} + audio_ff.{net.0.proj,net.2} × 48 transformer blocks (288 LoRA pairs total). Default rank 128 / alpha 128 / dropout 0.1, cosine LR schedule from 1e-4 with 500-step warmup over 10k steps.

To monitor training, set val_config: configs/val_config.example.yaml in your training YAML — src/validate.py is then spawned at every save step to generate one wav per speaker entry, so you can A/B listen during the run.

Inference with your trained LoRA

python src/inference.py \
  --lora /path/to/your/lora_step_5000.safetensors \
  --voice-sample reference.wav \
  --prompt 'A woman speaks warmly, "..."' \
  --output output.wav

Always load the LoRA at inference rather than pre-merging it — pre-merged checkpoints have produced degraded output in our runs.

Language

English.

License & acknowledgement

DramaBox is a Resemble AI fine-tune of LTX-2. Distributed under the LTX-2 Community License Agreement — see LICENSE. Thanks again to Lightricks for releasing the base model.

DramaBox (resemble-ai/DramaBox)

Searching Database…

505 ·GitHub
DramaBox
What is this?DramaBox is Resemble AI's expressive text-to-speech model with voice cloning, fine-tuned on top of Lightricks' LTX-2.3 audio model. The prompt itself controls speaker identity, emotion, delivery style, laughs, sighs, pauses and transitions, and an optional 10-second voice reference clones the target voice timbre.
Is this a hosted service or something I run myself?It's a self-hosted model: you run it via a warm server, a CLI, or a Gradio app. Code and weights auto-download from Hugging Face into a local cache on first run.
How do I get it running?Use the Quick Start section: run the warm server (recommended), the CLI, or the Gradio app. Model weights (DiT transformer, audio components, text encoder) are auto-downloaded from the Hugging Face repo on first run.
What hardware do I need to run it?Expect around 24 GB of peak VRAM, with roughly 2.5 seconds per generation on an H100. Optional voice-reference denoising (RE-USE) pulls extra dependencies that aren't installed by default.
Is there a limit on how long the generated speech can be?The underlying audio model was trained on clips of ~20 seconds or less; output stays usable up to ~45 seconds but quality drifts beyond that. For longer prompts, the server automatically chunks text and crossfades the pieces together.
Does voice-reference denoising work on macOS and Windows too?Not out of the box — the denoising dependencies (mamba-ssm, causal-conv1d) have no pre-built wheels on macOS or Windows and need matching CUDA on Linux, so they're skipped by default; without them it falls back to a slower pure-PyTorch path on supported Linux boxes.
What does a DramaBox prompt actually look like?A prompt pairs a speaker description with quoted dialogue and stage directions: sounds like laughs or sighs go inside quotes so the model voices them literally, while actions like 'He clears his throat' go outside the quotes as directions.
Are the generated audio files watermarked?Yes, every output from the inference script or server is watermarked with Resemble Perth, an imperceptible neural watermark that survives MP3 compression and common editing; it can be disabled for debugging with a flag.
Can I fine-tune my own voice or style on top of DramaBox?Yes, you can train a new LoRA on top of DramaBox itself using provided preprocessing scripts for several dataset formats and a Hugging Face Accelerate training script, without starting from the raw LTX-2.3 base.
What language does the model support?English only, per the README.
What are similar projects?bark, VoxCPM, OmniVoice, csm, dia, FireRedAudio, MOSS-TTS, WanGP
Which topics is it in?text-to-audio, audio-generation, voice-cloning, speech-synthesis, gradio, lora, tts, text-to-speech, fine-tuning, transformer, audio, nvidia
Creator
resemble-ai
Size
6.7 MB
Stars
505
Updated
CommitMay 23
Age
6mo
Commits · 90 Days