LANGUAGES

Twelve languages measured, twelve ahead.

Fewer tokens than 12 other tokenizers on every Tier 1 language, on real sentences.

FEATURED

Swahili result

FLORES+ and NTREX-128

Swahili needs 8.4% fewer tokens than GPT-4o on the same FLORES+ sentences.

Nzinga needs 36,583 tokens for 1,012 FLORES+ sentences. GPT-4o, the best other tokenizer, needs 39,922.

8.4%

fewer than GPT-4o

1.35x

FLORES+ premium

test sentences overlap training text

Read the result

TIER 1 LANGUAGES MEASURED

Swahili

Hausa

Yoruba

Zulu

Kinyarwanda

Shona

Igbo

Somali

Four more languages, measured the same.

5 LANGUAGES

Pixel mosaic cover image for the Hausa result

FLORES+

Latin script

Hausa

Hausa needs 6.6% fewer tokens than GPT-4o on FLORES+ devtest.

6.6%

fewer tokens

1.43x

token premium on FLORES+

NTREX-128 and FLORES+

Diacritics

Yoruba

Yoruba needs 24.7% fewer tokens than Aya Expanse on NTREX-128, though above the target.

Zulu needs 13.4% fewer tokens than GPT-4o, the best other tokenizer.

FLORES+ and NTREX-128

Latin

Kinyarwanda

Kinyarwanda premium falls to 1.56 times English, from 1.71 with GPT-4o.

8.4%

fewer tokens than GPT-4o

YORUBA

Still the fewest tokens of any tokenizer tested, though above the target.

FLORES+ devtest

Yoruba, premium of 1.98x

Count your own sentence in the browser.

Paste a sentence into the live comparison. Every figure here is read from a results file.

Try the tokenizer

Read the docs

~/african-tokenizer

zsh

$ bash scripts/fetch_data.sh

$ export HF_TOKEN=hf_••••••••

$ python scripts/premium_v2.py

✓ NTREX-128 · ✓ FLORES+ · ✓ baselines

● 5 results files · about 4 min