Works With
/
Cloudflare

Cloudflare

Live comparison on Pages; API built for Workers.

Hosting
Live
4.155.0
Compare
See results
Deploy
~/african-tokenizer
$
npx wrangler pages deploy demo/dist --project-name nzinga-tokenizer
✓ built python scripts/build_demo.py
✓ project nzinga-tokenizer
● live nzinga-tokenizer.pages.dev
Pixel mosaic band in violet and magenta across pale cells
Nzinga · hosting
Pages and Workers ·
Cloudflare
pages.project
Live comparison project
nzinga-tokenizer
pages.redeploy
Demo last redeployed on
29 Oct
workers.api
Staging API status
Not deployed

Overview

The live comparison runs on Cloudflare Pages. Text is tokenized in the browser and never leaves the device. The API is built for Workers; staging is not deployed yet.

What runs there

nzinga-tokenizer
pages
Live comparison, tokenized in the browser
nzingha-tokenizer
redirect
Old address, forwards to the new one
nzinga-network
pages
This site, with search indexing off
nzinga-api-staging
worker
Staging API, test data only, not deployed
storage
durable
Data saved to Durable Object storage after changes

Setup

01

Get a token

Use a token with Pages Edit or Workers Edit.

02

Build the demo

build_demo.py writes the comparison page with the current tokenizers.

03

Deploy it with wrangler

Pages deploy for the demo; wrangler deploy for the API.

Works well with

See all

NTREX-128

Benchmark

1,997 news sentences in 26 African languages.

Hugging Face

Datasets

The private release, with a model card, loads in transformers. Training text comes from FineWeb-2.

Expo

App

The Nzinga app runs in Expo Go, updated through EAS.

Count your own sentence in the browser.

Paste a sentence into the live comparison. Every figure here is read from a results file.

Try the tokenizer
Read the docs
~/african-tokenizer
zsh
$ bash scripts/fetch_data.sh
$ export HF_TOKEN=hf_••••••••
$ python scripts/premium_v2.py
✓ NTREX-128 · ✓ FLORES+ · ✓ baselines
● 5 results files · about 4 min