Pixel mosaic in Nzinga colours, one bright cell at its core

Integrators

For Integrators

Use the Nzinga tokenizer, try the live comparison, and see what the API and the app do.

Gemma 3

Apache 2.0 code

Use the Tokenizer

The tokenizer is Google's Gemma 3 tokenizer with 32,000 African-language tokens appended. Every original token keeps its ID. It loads with the Hugging Face tokenizers library.

tokenize.py

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer/gemma3-afx32k.json")

enc = tok.encode("Huku shirika la Save the Children likitela pamoja timu", add_special_tokens=False)

print(len(enc.ids), enc.tokens)

reproduce.sh

python -m venv .venv && .venv/bin/pip install -r requirements.lock

bash scripts/fetch_data.sh

.venv/bin/python scripts/leaderboard.py

.venv/bin/python -m pytest -q tests

Vocabulary

294,145 tokens

To use it with a Gemma 3 model in transformers, resize the embedding matrix to 294,145 rows and train the new rows. Until a model is trained, the new tokens carry no meaning for the model.

Note

The Hugging Face release is private. Code is Apache 2.0; the tokenizer files keep the Gemma Terms of Use until the planned rebuild on Gemma 4.

The Live Comparison

The live comparison shows how each tokenizer splits any text you paste, the token count, and the cost of sending it a million times at a price you set.

Hakuna kitu kama Brexit nzuri," aliongeza.

Swahili

Nzinga

10 tokens

GPT-4o

11 tokens

Gemma 4

13 tokens

GPT-4

16 tokens

NTREX-128 sentence 1680 · RESULTS.md

Text is tokenized in the browser and never leaves the device.

The API at a Glance

The Nzinga API runs on Cloudflare Workers. The server and the app share the same consent, invite, contacts and earnings rules.

nzinga api

private

Sign-in with a one-time code

built

Invites

built

Contacts matching with hashed numbers

built

Recordings and consent records

built

Review and earnings

built

Audio upload to encrypted storage

not built

Payouts through a payment partner

not built

Source: Nzinga README

Reference

The full reference will be in the developer documentation. Until its address is public, the source of record is the GitHub repository.

The App

The app is built with Expo. Without a server it runs in demo mode with realistic local data. Calling will need a native build later.

Demo sign-in code

482917

See Try the Demo for the QR code and link.

What Is Not Yet Measured

Fewer tokens is the measured claim. A better model is the hypothesis.

Note

Whether a model trained with Nzinga is more accurate has not been measured. The experiment is built and waiting for GPU time.