Gemma 3 extension pack
32,000 African word pieces appended to Gemma 3's tokenizer. Every token keeps its ID.
What changed
gemma3-afx32k.json: 294,145 tokens, with new merges learned between existing Gemma 3 tokens and each language weighted equally.
Input is composed to Unicode NFC before splitting, so text with separate tone marks tokenizes like standard text.
To use it with a Gemma 3 model, resize the embeddings to 294,145 rows and train them.
Release notes
The pack leaves English and French unchanged: any new merge that would apply to reference English or French text is discarded. A Llama 3.1 variant is built the same way, with smaller gains.
Questions about this release
The docs cover the tokenizer, the API and the app. For anything else, contact us through GitHub.
Count your first tokens in five minutes.
Paste any text into the live comparison. It is tokenized in your browser and stays there.