First prototype, corrected
The afro32k claim, 14 of 26, compared premiums. Raw tokens: 7 of 26.
What changed
A correction section in the README, and tests that recompute every headline figure.
Wins are counted on raw tokens. The prototype needed 12% more tokens for English, which flattered its premiums.
Use 7 of 26 for the prototype, and the current Nzinga figures for current results. Nothing else changes.
Release notes
The prototype was trained from scratch on 65,000 sentences. Its weak English count lowered every premium it reported, so 14 of 26 wins rested on premiums alone.
A second from-scratch tokenizer wins on 23 of 26 languages on raw tokens, but needs 27% more tokens for English. So Nzinga extends Gemma 3.
Questions about this release
The docs cover the tokenizer, the API and the app. For anything else, contact us through GitHub.
Count your first tokens in five minutes.
Paste any text into the live comparison. It is tokenized in your browser and stays there.