The challenge
Hausa text needs more tokens than its English source. With Gemma 3, the base Nzinga extends, the same 1,012 FLORES+ sentences take 1.73 times as many tokens as English. Hausa has 43.9 million first-language speakers (Ethnologue, 2019).
The change
Nzinga added 32,000 African word pieces to Gemma 3's tokenizer, trained on open web text in 29 languages, including 25.0 million characters of Hausa. Every Gemma 3 token kept its ID.
example: ntrex-128 #448
text: Mawaƙin Rap yana canza sunan - zuwa Ye.
nzinga: 13
gpt-4o: 14
gemma-4: 15
Thirteen tokens for this Hausa sentence, where GPT-4o needs fourteen and GPT-4 needs seventeen.
Example sentence
,
NTREX-128 sentence 448
,
Hausa
The result
Nzinga needs 6.6% fewer tokens than GPT-4o on FLORES+ and 7.1% fewer on NTREX-128. The premium falls to 1.43 times English, inside the 1.3 to 1.5 target.
