Hugging Face
The private release, with a model card, loads in transformers. Training text comes from FineWeb-2.
Datasets
Release
Private
Publish
~/african-tokenizer
$
python scripts/publish_hf.py --repo gtcx-protocol/nzinga-tokenizer
✓ model card written
✓ visibility private
● base google/gemma-3-4b-pt
Nzinga · results
release.vocab
Tokens in release
294,145
corpus.chars
Training text in 29 languages
620M chars
experiment.status
Model experiment on HF Jobs
Ready
Overview
Hugging Face holds the private release, gtcx-protocol/nzinga-tokenizer, with a model card. Loaded through transformers, the tokenizer class is set to the fast tokenizer so the pack loads verbatim with NFC. The training text, FineWeb-2, and FLORES+ come from Hugging Face.
What lives there
nzinga-tokenizer
repo
Private release with a model card
PreTrainedTokenizerFast
class
Loads the pack verbatim, with NFC kept
fineweb-2
dataset
Open web text the pack trained on
flores_plus
dataset
FLORES+ devtest, gated, for Tier 1
nzinga-experiment
jobs
Model experiment, dry-run end to end
Setup
01
Log in to Hugging Face
Set HF_TOKEN with write access, and access to Gemma 3.
02
Publish the release
Run publish_hf.py. The repository stays private unless you pass the --public flag.
03
Load with transformers
Transformers loads it as a fast tokenizer, with NFC kept.
Count your own sentence in the browser.
Paste a sentence into the live comparison. Every figure here is read from a results file.
$ bash scripts/fetch_data.sh
$ export HF_TOKEN=hf_••••••••
$ python scripts/premium_v2.py
✓ NTREX-128 · ✓ FLORES+ · ✓ baselines
● 5 results files · about 4 min