Build
~/african-tokenizer
$
python scripts/build_extension.py --base llama3_1
✓ merges learned 51,200
✓ tokens kept 32,232
● vocabulary 160,256
Nzinga · results
llama31.tokens
Tokens with Llama 3.1
3,412,466
pack.tokens
Tokens with the pack
2,480,461
languages.ahead
Ahead of the best global
15 of 26
Overview
The same method on Llama 3.1 appends 32,000 tokens to its 128,256. It needs 27.3% fewer tokens than Llama 3.1. Gains are smaller than on Gemma 3, because Llama 3.1 splits many African characters into raw bytes.
What it adds
llama31-afx32k.json
file
Llama 3.1 with the pack
base ids
kept
Llama 3.1 token IDs, all unchanged
new ids
added
32,000 African tokens after the 128,256 base
vocabulary
size
160,256 tokens in the extended tokenizer
ge'ez
bytes
Ge'ez characters split into raw bytes
Setup
01
Accept the Llama licence
Llama 3.1 is under the Llama 3.1 Community License.
02
Load the pack
Load llama31-afx32k.json in Python with the Hugging Face tokenizers library.
03
Resize the embeddings
Give the model 160,256 rows and train the new ones.
Count your own sentence in the browser.
Paste a sentence into the live comparison. Every figure here is read from a results file.
$ bash scripts/fetch_data.sh
$ export HF_TOKEN=hf_••••••••
$ python scripts/premium_v2.py
✓ NTREX-128 · ✓ FLORES+ · ✓ baselines
● 5 results files · about 4 min