The SOTA tokenization library on all languages, all models, all hardwares.
Our goal with tokenizers is to develop and maintain the industry's standard tokenization engine, making it the defacto place for everyone to contribute to the whole ecosystem.
Release candidate: v1.0.0.rc.x
As we are switching from 0.23 to v1.0.0, the current library does not ship all features. If you are worried, check the v1 features section for more details of what we are bringing back.
We will publish a blog about what breaking changes we introduced. We strived to keep them as small as possible.
Installation
pip install --pre tokenizers
For Rust, add the crate to your Cargo.toml:
[dependencies]
tokenizers = "1.0.0"
Training (train) and downloading from the Hub (http) are opt-in features. The crate's
Using the crate section lists every feature and how to
build a smaller binary.
Usage
>>> from tokenizers import Tokenizer
>>> tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B") # or .from_file(path)
>>> compiled = tokenizer.compile()
>>> compiled.encode("Hello, y'all! How are you 😁 ?")
Encoding(ids=[128000, 9906, 11, 379, 65948, 0, 2650, 527, 499, 27623, 223, 949], type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
>>> compiled.encode(("What is this?", "A pair.")).type_ids
[0, 0, 0, 0, 0, 1, 1, 1, 1]
>>> compiled.tokenize("Hello")
['<|begin_of_text|>', 'Hello']
>>> compiled.decode([15339, 1917])
'hello world'
>>> compiled.decode_tokens(compiled.encode("Hello"))
['<|begin_of_text|>', 'Hello']
Tokenizer holds the tokenizer.json config and can be edited (e.g. tokenizer.truncation = ...);
compile() builds the CompiledTokenizer that encodes and decodes.
Performances
Bindings
v1.0.0 roadmap:
As we work toward v1, we are gonna bring back the entire python API that allows transformers' style of interacting with a tokenizer object:
TODOS:
Python Bindings
- Bring training back: this should be easy, but we want the perfs gains to come as well.
- Batch encoding API with zero-copy views on encoded ids
- dlpack API for torch / JAX / numpy interop.
- Composition: edit components of the pipeline (
tokenizer.normalizer = [...]). The editable
Tokenizer spec and compile() have landed; per-component handles are next.
tokenizer.save to persist a modified or trained tokenizer on disk.role_to_token override.- Bring back offset output: this should be useful for people training.
- Windowing / stride in encode.
Rust core
bitnorm: more performance from optimized normalization, as this will bring breaking changes we want to make sure it gets to v1.- Windowing (truncation with stride).
- Improve training performance
- Improve decode performance
- Offset tracking
- Padding: 2D flat buffer
- Wordcache:
shrn vs emulated movemask
- Fix ModernBERT / OLMo load failure ("64-bit hash collision").
- Unroll more regex: we need to cover the most used ones, we might have missed one or two.
Other
- C & C++ bindings for loading, encode and decode APIs
After 1.0.0
- Improve / rewrite the
bitcannon crate
- Java, Go, Swift... bindings, supported by the C bindings
tk-devices — exploratory GPU encoding and batch decoding that keeps text and ids on the
device: upload the vocabulary once, compute output positions in parallel, gather the bytes on the
GPU. An optional component aimed at large batches, subject to prototyping and measurement.
Crate details
Crate size
Sub-crates
The models (BPE, Unigram, WordPiece, WordLevel) and the full pipeline: Normalizer,
PreTokenizer, Model, PostProcessor, Decoder.
Why separate: you don't need all features to serve a model in production settings. Splitting it from training
means a serving binary never links a trainer, a corpus reader, legacy converters, etc.
Unicode tag classification plus pre-tokenization as a bitstream program. Follows Interleaved Bitstream Execution for Multi-Pattern Regex Matching on GPUs
(MICRO'25, 10.1145/3725843.3756052) but we added our own algorithm to make sure we don't need a parabix engine.
Characters are first classified into "tags", which use a sparse table generated when compiling bitcannon. This means there are 0 depencencies to the heavy unicode tables.
Ships byte-exact grammars for gpt2 / ByteLevel, cl100k, o200k, tekken, deepseek and kimi-k2.
The classes or tags each have a high and low nibble (encoded on a u8), and carry refinement when needed. Using u8 tags allows us to leverage less bitsream, meaning have a simpler / faster pre tokenization.
Why separate: we just thought it might be useful to some people someday. As far as we know, this implementation is the fastest for every single pretokenizer regex on CPU.
from_json_file turns a canonical tokenizer.json into a PipelineTokenizer.
Why separate: this just makes it optional for some usecases!
canonicalize_file rewrites legacy a tokenizer.json written by any older version of this library into
the canonical form the reader accepts. A pure JSON→JSON rewrite; it depends on nothing but
std::path and serde_json.
Why separate: every config ever published stays readable without the runtime carrying a decade
of compatibility branches. But, the legacy code amounted to quite a lot of the final crate size. Disabling it allows on device builds to be smaller.
The Trainer trait, every concrete *Trainer, TrainerWrapper and the Trainable extension.
여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗
