Gemma 3
The base of the current pack: 32,000 African tokens appended.
Models
Current
afx32k
Build
~/african-tokenizer
$
python scripts/build_extension.py --base gemma3
✓ merges learned 51,200
✓ pack written 32,000 tokens
● vocabulary 294,145
Nzinga · results
gemma3.vocab
Base vocabulary size
262,145
nzinga.vocab
With the pack
294,145
tier1.premium
Average Tier 1 premium
1.55x
Overview
Nzinga extends Gemma 3's tokenizer: 32,000 African word pieces are appended, every Gemma 3 token keeps its ID, and input is converted to Unicode composed form first. Gemma 4 splits text identically; a port is planned.
What it adds
gemma3-afx32k.json
file
The tokenizer with the 32,000-token pack
base ids
kept
Gemma 3 token IDs, all unchanged
new ids
added
32,000 African tokens after the base
normalizer
nfc
Unicode composed form applied before splitting any text
embeddings
rows
294,145 rows after resizing the model
Setup
01
Accept the Gemma terms
Gemma 3 is gated on Hugging Face; accept its Terms of Use first.
02
Load the pack
Load gemma3-afx32k.json in Python with the Hugging Face tokenizers library.
03
Resize the embeddings
Give the model 294,145 rows and train the new ones.
Count your own sentence in the browser.
Paste a sentence into the live comparison. Every figure here is read from a results file.
$ bash scripts/fetch_data.sh
$ export HF_TOKEN=hf_••••••••
$ python scripts/premium_v2.py
✓ NTREX-128 · ✓ FLORES+ · ✓ baselines
● 5 results files · about 4 min