sentencepiece README

The project's README from github.com/google/sentencepiece. As one markdown file: README.md.

SentencePiece

Build C++ Build Wheels GitHub Issues PyPI - Python Version PyPI version PyPi downloads Contributions welcome License SLSA 3 Documentation

SentencePiece is a fast, lightweight, and unsupervised text tokenizer and detokenizer designed for neural network-based text generation systems (such as Large Language Models) where the vocabulary size is fixed prior to training.

It implements subword units—including Byte-Pair-Encoding (BPE) [Sennrich et al.] and the unigram language model [Kudo.]—with the ability to train directly from raw sentences. By treating input text as a raw sequence of Unicode characters, SentencePiece enables a purely end-to-end, language-independent pipeline that completely eliminates the need for language-specific pre- or post-processing.

This is not an official Google product.

Quick Start (Python)

SentencePiece provides an easy-to-use Python module. Install it via pip:

pip install sentencepiece

Basic Example

Here is how to train a model, encode text into tokens/IDs, and decode them back to the original string:

import sentencepiece as spm

# 1. Train a model directly from a raw text file.
# (No pre-tokenization or language-specific preprocessing required!)
spm.SentencePieceTrainer.train(
    input='data/botchan.txt', 
    model_prefix='m', 
    vocab_size=1000
)

# 2. Load the trained model.
sp = spm.SentencePieceProcessor(model_file='m.model')

# 3. Encode raw text into subword pieces (strings) or vocabulary IDs (integers).
text = "I saw a girl with a telescope."
pieces = sp.encode(text, out_type=str)
ids = sp.encode(text, out_type=int)

print(f"Pieces: {pieces}")
# Output: ['▁I', '▁saw', '▁a', '▁girl', '▁with', '▁a', '▁', 'te', 'le', 's', 'c', 'o', 'pe', '.']

print(f"IDs:    {ids}")
# Output: [9, 459, 11, 939, 44, 11, 4, 142, 82, 8, 28, 21, 132, 6]

# 4. Decode IDs or pieces back into the original text.
# The reconstruction is completely lossless and reversible!
print(sp.decode(ids))
# Output: "I saw a girl with a telescope."

print(sp.decode(pieces))
# Output: "I saw a girl with a telescope."

Why SentencePiece?

1. Reversible & Lossless Tokenization (Whitespace as a Basic Symbol)

Traditional tokenizers drop whitespace information (e.g., treating Tokenize("World.") identically to Tokenize("World .")), making detokenization ambiguous and language-dependent.

SentencePiece treats the input text as a raw sequence of Unicode characters. It escapes whitespaces with a meta-symbol ▁ (U+2581) and includes it in the tokenization. This design ensures that detokenization is a simple, lossless string join operation, entirely independent of the language:

# Lossless detokenization
original_text = "".join(pieces).replace("▁", " ")

2. Purely Data-Driven & Language-Independent

SentencePiece trains tokenization and detokenization models directly from raw sentences. It does not require language-specific pre-tokenizers (such as Moses, MeCab, or KyTea). This makes it highly effective for languages without explicit word boundaries, such as Chinese, Japanese, and Thai.

3. Subword Regularization & BPE-Dropout

To improve the robustness and accuracy of translation and language models, SentencePiece supports on-the-fly subword sampling during training. By sampling different segmentations for the same input text (Subword Regularization for Unigram, BPE-Dropout for BPE), it virtually augments your training data and makes the model more resilient to spelling variations and noise.

# Sample different segmentations on-the-fly
for _ in range(3):
    print(sp.encode('New York', out_type=str, enable_sampling=True, alpha=0.1, nbest_size=-1))
# May output:
# ['▁', 'N', 'e', 'w', '▁York']
# ['▁New', '▁York']
# ['▁New', '▁Y', 'o', 'r', 'k']

4. Fast, Lightweight, and Self-Contained

Performance Benchmark (SentencePiece vs. Hugging Face Fast)

Benchmark Setup

1. Unigram Model: T5-base (32k vocab)

Tokenizer 1 Thread 2 Threads 4 Threads 8 Threads 16 Threads 24 Threads
SentencePiece 27.41 43.83 71.62 102.08 123.33 127.60
Hugging Face Fast 3.78 7.15 12.45 20.33 27.00 31.49

2. BPE Model: Gemma 3 (256k vocab)

Tokenizer 1 Thread 2 Threads 4 Threads 8 Threads 16 Threads 24 Threads
SentencePiece 7.44 12.82 23.03 36.66 48.65 52.43
Hugging Face Fast 3.66 6.37 10.45 15.54 21.05 20.48

Why performance does not scale linearly:

While the core tokenization (C++ or Rust) runs in parallel, the final step of converting the native results (C++ vector of vectors or Rust vector of vectors) into Python objects (list[list[int]] or list[Encoding]) is sequential and must be done on Python's main thread (GIL-locked). At high thread counts, this single-threaded serialization step becomes the dominant bottleneck, capping the scaling performance.

For the detailed analysis and single-thread reference comparison, see Performance Benchmark Details.

To run these benchmarks yourself, see the reproduction instructions and scripts.

Documentation & Resources

For detailed guides, API references, and advanced usage, please refer to the following resources:

License

SentencePiece is licensed under the Apache 2.0 License.

sentencepiece (google/sentencepiece)

Searching Database…

12.1K ·GitHub
Sentencepiece
What is this?SentencePiece is a fast, lightweight, unsupervised text tokenizer and detokenizer built for neural-network text generation systems like LLMs, where vocabulary size is fixed before training. It implements BPE and the unigram language model as subword units and trains directly from raw sentences.
Is this a Python library, or a standalone CLI tool?Both: it's a C++ core library exposed through an easy-to-use Python module, plus a command-line interface for training and running tokenizers, with the same .model file usable from C++, Python, or Go.
How do I get it running?Install the Python module with pip, then train a model on raw sentences and use it to encode text into tokens/IDs and decode back to the original string.
Do I need anything besides Python to use it?For the Python workflow, just pip install is enough. Building the C++ library or CLI from source requires CMake or Bazel instead.
Is this an official Google product?No, the README explicitly states it is not an official Google product, so support and roadmap follow normal open-source norms rather than official backing.
How does it handle whitespace during tokenization?It escapes whitespace with the meta-symbol ▁ (U+2581) and keeps it in the token stream, so detokenization becomes a simple, lossless string join instead of a language-dependent heuristic.
Can I reuse the same tokenizer model across C++, Python, and Go?Yes, each generated .model file is self-contained with all normalization rules, vocabulary mapping, and segmentation logic, guaranteeing identical tokenization results in any language binding.
What is subword regularization?It's on-the-fly sampling of different subword segmentations for the same input during training (Subword Regularization for unigram, BPE-Dropout for BPE), which augments the training data and makes models more robust to spelling noise.
Which kinds of languages benefit most from using SentencePiece?Languages without explicit word boundaries, like Chinese, Japanese, and Thai, benefit most since SentencePiece trains purely from raw text and needs no language-specific pre-tokenizers such as Moses, MeCab, or KyTea.
What are similar projects?Chinese-Word-Vectors, spaCy, jeeves, minGPT, Hy-MT2, llama3-from-scratch, DMI, tokenizers
Which topics is it in?tokenization, natural-language-processing, neural-network, nlp, artificial-intelligence, huggingface, c++, benchmark, performance, deep-learning, research, machine-learning
Creator
google
Size
47.2 MB
Stars
12.1K
Updated
CommitOct 16Releasev0.2.2Jul 12
Age
9y
Commits · 90 Days