Open research implementation
JSON Tokenizer
Test whether a repeated JSON schema deserves its own vocabulary.
A structure-aware tokenizer that learns JSON keys and value fragments. Train it on your schema, inspect the token sequence, and compare token counts and round-trip behavior on held-out examples.
What you can do with it
Who it's for: Researchers building structured-data models or custom encoding pipelines.
A tokenizer trained on your data, with a measured compression and correctness comparison.
Your first useful result
- 01
Train a local vocabulary
From the repository, replace the input path with a JSON or JSONL training file. Keep evaluation examples out of the training data.
python -m json_tokenizer.cli train data/training_corpus.jsonl -o ./model --vocab-size 4096 - 02
Count a held-out example
Replace data.json with your evaluation file and inspect its token count.
python -m json_tokenizer.cli count data.json -m ./model - 03
Verify the round trip
Use JSONTokenizer.encode and decode from the README to compare decoded values with the original objects. Benchmark prose-heavy data separately from repetitive schemas.
Know the limits
- A custom token vocabulary does not reduce a hosted model API bill unless the model actually uses that vocabulary.
- The published results include regressions on prose-heavy and function-calling datasets. The Python implementation is slower than tiktoken.
- Round-trip object equality and byte-identical source text are different checks; verify the one your application needs.