11.2K ·GitHub
What is this?
Tokenizers is a library implementing today's most-used tokenizers (BPE, WordPiece, Unigram) for NLP, built in Rust for speed and exposed to other languages via bindings. It's made by Hugging Face and powers tokenization for transformer models.
Tokenizers is a library implementing today's most-used tokenizers (BPE, WordPiece, Unigram) for NLP, built in Rust for speed and exposed to other languages via bindings. It's made by Hugging Face and powers tokenization for transformer models.Is this a standalone app or a library I import into code?It's a library: a Rust core with bindings for Python, Node.js, and a community Ruby binding, meant to be imported into your own training/inference code.
It's a library: a Rust core with bindings for Python, Node.js, and a community Ruby binding, meant to be imported into your own training/inference code.How do I install and use it?Install from source or via the released package for your language (e.g. pip for Python), then pick a model type like BPE, WordPiece, or Unigram, instantiate a tokenizer, and train it on your files in just a couple lines of code.
Install from source or via the released package for your language (e.g. pip for Python), then pick a model type like BPE, WordPiece, or Unigram, instantiate a tokenizer, and train it on your files in just a couple lines of code.Do I need Rust installed to use the Python bindings?No separate Rust toolchain is required for end users — the released Python/Node packages ship precompiled, you only need source build tools if compiling from source.
No separate Rust toolchain is required for end users — the released Python/Node packages ship precompiled, you only need source build tools if compiling from source.Does performance depend on my hardware?Yes, benchmarked speed numbers can vary depending on hardware; the README gives figures from a specific AWS g6 instance as a reference point, not a universal guarantee.
Yes, benchmarked speed numbers can vary depending on hardware; the README gives figures from a specific AWS g6 instance as a reference point, not a universal guarantee.Who is this built for?It targets both researchers experimenting with tokenization and production teams needing fast, reliable text preprocessing for language models.
It targets both researchers experimenting with tokenization and production teams needing fast, reliable text preprocessing for language models.How fast is tokenization really?The README states it takes less than 20 seconds to tokenize a gigabyte of text on a server's CPU, thanks to the Rust implementation.
The README states it takes less than 20 seconds to tokenize a gigabyte of text on a server's CPU, thanks to the Rust implementation.Can I trace tokens back to the original text?Yes, normalization comes with alignment tracking, so you can always retrieve the exact part of the original sentence a given token corresponds to.
Yes, normalization comes with alignment tracking, so you can always retrieve the exact part of the original sentence a given token corresponds to.Does it handle padding and truncation for me?Yes, it does all the needed pre-processing: truncating, padding, and adding the special tokens your model requires.
Yes, it does all the needed pre-processing: truncating, padding, and adding the special tokens your model requires.Which tokenizer algorithms are supported?You can choose between Byte-Pair Encoding (BPE), WordPiece, or Unigram models when instantiating a tokenizer.
You can choose between Byte-Pair Encoding (BPE), WordPiece, or Unigram models when instantiating a tokenizer.What are similar projects?Chinese-BERT-wwm, LLMs-from-scratch, bert, RWKV-LM, LoRA, Transformers-Tutorials, StableLM, minGPT
Chinese-BERT-wwm, LLMs-from-scratch, bert, RWKV-LM, LoRA, Transformers-Tutorials, StableLM, minGPTWhich topics is it in?tokenizer, bert, language-model, natural-language-processing, transformers, gpt, nlp, huggingface, performance, rust, deep-learning, research
tokenizer, bert, language-model, natural-language-processing, transformers, gpt, nlp, huggingface, performance, rust, deep-learning, research