Twelve languages on FLORES+
Akan and Lingala added. Fewer tokens than 12 other tokenizers on all 12.
What changed
Akan and Lingala, trained on FineWeb-2 text and 24,502 Google WAXAL speech transcripts, read as text only.
Both packs were rebuilt with stricter French protection, because Lingala and Akan web text borrow French words. It cut the Igbo lead from 7.9% to 4.3%.
None. The Tier 1 average premium is now 1.55 times English, and 6 of the 12 languages meet the 1.3 to 1.5 target.
Release notes
Tier 1 is now 12 of 12 on FLORES+: fewer tokens than every other tokenizer tested, in every language, by 4% to 28%.
No FLORES+ devtest sentence in any Tier 1 language shares a 13-word sequence with the training text.
Questions about this release
The docs cover the tokenizer, the API and the app. For anything else, contact us through GitHub.
Count your first tokens in five minutes.
Paste any text into the live comparison. It is tokenized in your browser and stays there.