The challenge
Yoruba writes tone marks, and 1,311 of 1,997 NTREX-128 sentences write them as separate characters. With Gemma 3, Yoruba takes 3.17 times the tokens of English.
The change
Nzinga converts input to the standard composed form before splitting, and added 32,000 African word pieces trained on open web text, including 25.0 million characters of Yoruba.
normalize: nfc
ntrex-128:
nzinga: 115,879
aya-expanse: 153,841
flores+:
nzinga: 53,701 · gpt-4o: 58,238
Still the fewest tokens of any tokenizer tested, though above the target.
FLORES+ devtest
,
premium 1.98x
,
Yoruba result
The result
Nzinga needs 24.7% fewer tokens than Aya Expanse on NTREX-128 and 7.8% fewer than GPT-4o on FLORES+. The FLORES+ premium is 1.98 times English, still above the 1.3 to 1.5 target.
