Every Nzinga figure is measured on sentences the tokenizer never saw in training. That promise is easy to make and harder to keep: web text copies itself, and test sentences turn up in training data more often than anyone expects.
Why held-out text matters
A tokenizer trained on its own test sentences learns them as whole pieces and looks better than it is. The figure then measures memory, not efficiency. A fair comparison needs text that every tokenizer meets for the first time, and a check that proves it, on every release.
Two test sets
Nzinga is measured on public test sets, professionally translated into each language and kept out of training, plus one more for models:
- FLORES+ devtest: 1,012 sentences per language, from Wikipedia-family sites
- NTREX-128: 1,997 news sentences in each of 26 African languages
- Belebele: 488 passages per language, kept for the model experiment
Training text is checked against all three.
swh_Latn 1,012 sentences
13-word overlap: 0
hau_Latn 1,012 sentences
13-word overlap: 0
21 of 21 languages: 0The 13-word overlap check
Any training document that shares a 13-word sequence with any NTREX-128 or FLORES+ devtest sentence, in any language, is dropped. The check runs again on every release: 0 of 1,012 sentences in each of the 21 FLORES+ languages share one.
“Neither NTREX-128 nor FLORES+ was ever used in training.”
What it does not show
Held-out text makes the token counts fair. It does not show that a model trained with Nzinga is more accurate. That needs a trained model scored on bits per character and Belebele, and the experiment is built and waiting for GPU time.
