AI models read, bill and limit text in tokens, the word pieces a tokenizer splits text into. For the same meaning, African languages need more tokens than English. That difference is the token premium.
Tokens, not words
A premium is the tokens a language needs divided by the tokens English needs for the same sentences, with the same tokenizer. At 1.5 times, a message needs 50% more tokens, and so 50% more cost and time.
Where it shows
It is paid three ways on every query: in cost at any per-token price, in context, because less text fits a window, and in speed. The arithmetic is short:
premium = tokens(lang) / tokens(eng)
# FLORES+ devtest, Swahili, 1,012 sentences
# GPT-4o 1.49x · Nzinga 1.35x
# 128k window: 71,900 -> 79,100 words“A tokenizer is fixed into every model trained with it.”
What twelve tokenizers charge today
Across the 12 Tier 1 languages on FLORES+, Gemma 3 and Gemma 4 average 1.97 times English, and GPT-4 2.71. Nzinga averages 1.55, and 6 of the 12 meet the 1.3 to 1.5 target.
Fixing it at the tokenizer
Cutting the premium at the tokenizer is among the least expensive ways to make AI in African languages cheaper and faster.
