Works With
/
NTREX-128

NTREX-128

1,997 news sentences in 26 African languages.

Benchmark
Open
1,997
Compare
See results
Fetch
~/african-tokenizer
$
bash scripts/fetch_data.sh
✓ cloned MicrosoftTranslator/NTREX
✓ languages 26 African
● also fetched MAFAND-MT
Pixel mosaic band in violet and magenta across pale cells
Nzinga · results
1,997 sentences, news ·
NTREX-128
languages.ahead
Languages ahead of every baseline
24 of 26
mean.premium
Mean premium against English
1.52x
pooled.saving
Fewer tokens than the best
12.8%

Overview

NTREX-128 is Microsoft's news test set: 1,997 sentences, professionally translated into each language. Nzinga uses it for 26 African languages. The two gaps, Hassaniya Arabic and Tachelhit, are in Arabic script, with no open training text in that script.

What the set measures

sentences
count
1,997 news sentences in each language
languages
count
26 African languages, plus English
baselines
count
Nine global tokenizers, twelve in v2
overlap
check
Training documents sharing 13 words: 0
leaderboard
file
Every count, in results/leaderboard.json, the source of truth

Setup

01

Clone the test set

fetch_data.sh clones the NTREX-128 repository locally.

02

Run the benchmark

Run premium_v2.py. It counts tokens for every tokenizer, language and sentence in about four minutes.

03

Check the claims

pytest compares every README figure with the results files.

Works well with

See all

FLORES+

Datasets

Tier 1 test set: 1,012 sentences per language.

Cloudflare

Hosting

Live comparison on Pages; API built for Workers.

Hugging Face

Datasets

The private release, with a model card, loads in transformers. Training text comes from FineWeb-2.

Count your own sentence in the browser.

Paste a sentence into the live comparison. Every figure here is read from a results file.

Try the tokenizer
Read the docs
~/african-tokenizer
zsh
$ bash scripts/fetch_data.sh
$ export HF_TOKEN=hf_••••••••
$ python scripts/premium_v2.py
✓ NTREX-128 · ✓ FLORES+ · ✓ baselines
● 5 results files · about 4 min