Our first prototype was reported as beating the best of nine global tokenizers on 14 of 26 languages. That compared premiums. On raw tokens for the same sentences, it was 7.
What we first reported
The first prototype, afro32k, was trained from scratch on 65,000 sentences. The brief compared each premium with the best global tokenizer, language by language, and counted a win wherever its premium was lower.
Where the premium misleads
A premium divides a language's tokens by that same tokenizer's English tokens. So a weak English count flatters every premium:
- The prototype needed 12% more tokens for English
- That larger English count lowered every premium the prototype reported
- Its Yoruba 1.75x did not survive
Premiums still matter: they are the cost multiple a user of that tokenizer pays.
Count raw tokens instead
Wins are now counted on raw token counts for the same sentences. A test recomputes every headline figure:
.venv/bin/python -m pytest -q tests
# README figures against results/What changes in our reporting
Use 7 of 26 for the prototype. Current results are wins on raw tokens: 24 of 26 on NTREX-128 and 21 of 21 on FLORES+.
“Use 7 of 26 for that prototype, not 14.”
