Nzinga is not a new tokenizer. It is Gemma 3's tokenizer with 32,000 African word pieces added after it, and every existing token keeps its ID. That choice decides almost everything else.
Why extend, not replace
Our second from-scratch tokenizer beat every global tokenizer on 23 of 26 languages on raw tokens. It also needed 27% more tokens for English than the best of them. Extending an established tokenizer avoids that trade-off.
What changes
- 32,000 new African word pieces
- Input composed to Unicode NFC before it is split
- Special tokens keep IDs
Load it like any other tokenizer file:
Tokenizer.from_file("tokenizer/gemma3-afx32k.json")English and French, protected
Any new merge that would apply to reference English or French text is discarded. Gemma 3's merges take priority, so text without the new pairs tokenizes as before.
“English and French change by one to three tokens.”
Limits
French still needs 5.3% more tokens than the most efficient tokenizer tested, because Nzinga keeps French identical to Gemma 3. The pack also carries the Gemma Terms of Use until it is rebuilt on Gemma 4.
