csm README

The project's README from github.com/SesameAILabs/csm. As one markdown file: README.md.

CSM

2025/05/20 - CSM is availabile natively in Hugging Face Transformers 🤗 as of version 4.52.1, more info available in our model repo

2025/03/13 - We are releasing the 1B CSM variant. The checkpoint is hosted on Hugging Face.

CSM (Conversational Speech Model) is a speech generation model from Sesame that generates RVQ audio codes from text and audio inputs. The model architecture employs a Llama backbone and a smaller audio decoder that produces Mimi audio codes.

A fine-tuned variant of CSM powers the interactive voice demo shown in our blog post.

A hosted Hugging Face space is also available for testing audio generation.

Requirements

Setup

git clone [email protected]:SesameAILabs/csm.git
cd csm
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Disable lazy compilation in Mimi
export NO_TORCH_COMPILE=1

# You will need access to CSM-1B and Llama-3.2-1B
huggingface-cli login

Windows Setup

The triton package cannot be installed in Windows. Instead use pip install triton-windows.

Quickstart

This script will generate a conversation between 2 characters, using a prompt for each character.

python run_csm.py

Usage

If you want to write your own applications with CSM, the following examples show basic usage.

Generate a sentence

This will use a random speaker identity, as no prompt or context is provided.

from generator import load_csm_1b
import torchaudio
import torch

if torch.backends.mps.is_available():
    device = "mps"
elif torch.cuda.is_available():
    device = "cuda"
else:
    device = "cpu"

generator = load_csm_1b(device=device)

audio = generator.generate(
    text="Hello from Sesame.",
    speaker=0,
    context=[],
    max_audio_length_ms=10_000,
)

torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate)

Generate with context

CSM sounds best when provided with context. You can prompt or provide context to the model using a Segment for each speaker's utterance.

NOTE: The following example is instructional and the audio files do not exist. It is intended as an example for using context with CSM.

from generator import Segment

speakers = [0, 1, 0, 0]
transcripts = [
    "Hey how are you doing.",
    "Pretty good, pretty good.",
    "I'm great.",
    "So happy to be speaking to you.",
]
audio_paths = [
    "utterance_0.wav",
    "utterance_1.wav",
    "utterance_2.wav",
    "utterance_3.wav",
]

def load_audio(audio_path):
    audio_tensor, sample_rate = torchaudio.load(audio_path)
    audio_tensor = torchaudio.functional.resample(
        audio_tensor.squeeze(0), orig_freq=sample_rate, new_freq=generator.sample_rate
    )
    return audio_tensor

segments = [
    Segment(text=transcript, speaker=speaker, audio=load_audio(audio_path))
    for transcript, speaker, audio_path in zip(transcripts, speakers, audio_paths)
]
audio = generator.generate(
    text="Me too, this is some cool stuff huh?",
    speaker=1,
    context=segments,
    max_audio_length_ms=10_000,
)

torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate)

FAQ

Does this model come with any voices?

The model open-sourced here is a base generation model. It is capable of producing a variety of voices, but it has not been fine-tuned on any specific voice.

Can I converse with the model?

CSM is trained to be an audio generation model and not a general-purpose multimodal LLM. It cannot generate text. We suggest using a separate LLM for text generation.

Does it support other languages?

The model has some capacity for non-English languages due to data contamination in the training data, but it likely won't do well.

Misuse and abuse ⚠️

This project provides a high-quality speech generation model for research and educational purposes. While we encourage responsible and ethical use, we explicitly prohibit the following:

By using this model, you agree to comply with all applicable laws and ethical guidelines. We are not responsible for any misuse, and we strongly condemn unethical applications of this technology.

Authors

Johan Schalkwyk, Ankit Kumar, Dan Lyth, Sefik Emre Eskimez, Zack Hodari, Cinjon Resnick, Ramon Sanabria, Raven Jiang, and the Sesame team.

csm (SesameAILabs/csm)

Searching Database…

14.7K ·GitHub
Csm
What is this?CSM (Conversational Speech Model) is a speech generation model from Sesame that turns text and audio input into spoken audio by producing RVQ audio codes, using a Llama backbone plus a smaller audio decoder.
Is this a chatbot or just a voice generator?It is purely a speech generation model, not a conversational chatbot: it cannot generate text, so you need to pair it with a separate LLM if you want an actual conversation.
How do I get it running?Clone the repo and run the quickstart script, which generates a conversation between two characters from text prompts; for custom use you call the model directly to generate a single sentence or generate speech with added context via Segment objects.
What do I need before I can run it?A CUDA-compatible GPU (tested on CUDA 12.4/12.6), Python 3.10 recommended, ffmpeg for some audio operations, and Hugging Face access to the Llama-3.2-1B and CSM-1B checkpoints. On Windows, you must use pip install triton-windows instead of the normal triton package.
Does it come with ready-made voices or support other languages?No - it's a base model not fine-tuned to any specific voice, so you get a random voice unless you supply a prompt/context, and non-English languages are only weakly handled because of incidental data contamination rather than intentional training.
Does it really need a dedicated GPU to work?Yes, the project requires a CUDA-compatible GPU; it has only been verified on CUDA 12.4 and 12.6.
Why does CSM use a Llama backbone for audio generation?CSM pairs a Llama-based backbone with a smaller audio decoder that outputs Mimi audio codes, letting it condition speech generation on text and prior audio context the way a language model predicts tokens.
Can I use CSM to clone a real person's voice?No - the project explicitly prohibits impersonation or fraud, meaning generating speech that mimics a real individual without their consent violates its stated usage policy.
Can I try CSM without installing anything locally?Yes, a hosted Hugging Face Space is available for testing audio generation, and a fine-tuned variant of CSM powers the interactive voice demo shown in Sesame's blog post.
What are similar projects?bark, dia, ChatTTS, Speech, fish-speech, DramaBox, FireRedAudio, VoxCPM
Which topics is it in?conversational-ai, audio-generation, speech-synthesis, llama, voice, tts, text-to-speech, transformer, audio, artificial-intelligence, inference, nvidia
Creator
SesameAILabs
Size
<0.1 MB
Stars
14.7K
Updated
CommitMay 27
Age
1y
Commits · 90 Days