v2 results: 21 of 21 FLORES+ languages
Fewer tokens than 12 baselines, Gemma 4 included, on 21 of 21 FLORES+ languages and 24 of 26 NTREX-128.
What changed
Three new baselines: Google Gemma 4, Alibaba Qwen 3.8 and OpenAI gpt-oss, measured from their public tokenizer files.
Wins are counted on raw tokens for the same sentences. Pooled saving against the best baseline per language: 13.5% on FLORES+.
None for the tokenizer. Gemma 4 changes 6,207 token IDs, so using it needs a rebuilt pack.
Release notes
Nzinga needs fewer tokens than every baseline on 21 of 21 FLORES+ languages and 24 of 26 on NTREX-128. The exceptions, Hassaniya Arabic and Tachelhit, have no open training text.
Gemma 4 splits every language in both sets exactly as Gemma 3 does.
Questions about this release
The docs cover the tokenizer, the API and the app. For anything else, contact us through GitHub.
Count your first tokens in five minutes.
Paste any text into the live comparison. It is tokenized in your browser and stays there.