A few months of ternary experiments

Where ternary {−1, 0, +1} weights are the right tool for small, on-device models — and where they aren't.

Ternary weights — every weight forced to −1, 0, or +1, about 1.58 bits each — were shown to work at scale by BitNet b1.58: a 3B ternary model matching FP16 LLaMA at 3.55× less memory. That the idea works was settled before I touched it. The question I care about is the practical follow-on: for the small, on-device models I actually build, where is ternary the right tool — and where isn't it?

This is a log of a few months answering that — from a wave of tiny single-purpose grammar models, to a 10 MB date parser and sub-megabyte web-element classifiers that ship on the Neural Engine, to training a general ternary base from scratch up a cost-controlled "rung ladder". The recurring finding: quantization-aware training costs little or nothing at this scale, and the pattern that carried through is one frozen ternary base plus small task-specific LoRA adapters.

1. The starting point: BitNet

This began with BitNet b1.58 (Ma et al., 2024). The reported result: train a transformer whose every weight is one of three values — −1, 0, or +1 — and it stays competitive with a full-precision model of the same size. At 3B parameters their ternary model matched an FP16 LLaMA in both perplexity and zero-shot accuracy while using 3.55× less memory. Three values is about 1.58 bits per weight, the information content of a three-sided coin.

The basic question — does ternary work? — was answered before I touched it. The straight-through estimator that makes quantized training possible goes back to Bengio et al. (2013); binary-weight training to BinaryConnect and XNOR-Net (2015–16); ternary-with-a-scale to Ternary Weight Networks (Li & Liu, 2016) and Trained Ternary Quantization (Zhu et al., 2017); and ternary at LLM scale to BitNet b1.58 (2024). My interest is the practical follow-on: for the small, on-device things I build, where is ternary the right tool?

The mechanics, briefly

A normal layer is the matrix multiply y = W·x over float weights. Ternary forces each weight to −1, 0, or +1 and factors out one shared scale per row, W ≈ α·T. The multiplies become additions — multiplying by −1/0/+1 is a sign flip, a skip, or a copy — and storage drops by roughly 10–16×. To select which weights go to zero, I reuse the rule from Ternary Weight Networks: zero anything below 0.7·mean(|W|), keep the sign of the rest, share one scale. This happens during training (quantization-aware training), not as a rounding step afterward — the straight-through estimator lets gradients flow to a full-precision shadow copy while the forward pass sees the snapped weights.

2. First wave: many small grammar models worked

I started broad: a set of tiny ternary models, each handling one narrow linguistic job. A model for verb tense, one for subject–verb agreement, articles, pronouns, punctuation, casing, part-of-speech, spelling, contractions, noun number, dates. All used the same standard architecture — RMSNorm, SwiGLU, rotary embeddings — all trained from scratch on synthetic data generated by a script, all ternary via QAT. The goal was to see whether enough of these specialists, stacked together, could do full grammar correction on-device — each one small enough that running a dozen at once stays within a phone’s budget.

Across the wave the pattern held: on tasks this narrow, ternary QAT cost little or nothing against full precision — enough to push the recipe somewhere it would be used.

3. A date parser in 10 MB: date-lm worked

The most developed of the grammar models, and the one I leaned on most, parses dates. The task: turn a phrase like “next friday at 3pm” or “end of last quarter” into an ISO 8601 timestamp, given the current time as context. The macOS built-in, NSDataDetector, handles around 72% of realistic phrasings and fails on sub-day relatives, holidays, recurrence, and fiscal periods.

date-lm is a ~10M-parameter decoder-only seq2seq model (6 layers, 384-dim, 6 heads, an ~800-token byte-level BPE vocab). The training set is 100K synthetic pairs from a generator that composes twenty-plus expression types with conversational filler — no collected data, just a grammar — and training takes about 30 minutes on one rented A100. The ternary (QAT) version matches the full-precision model on exact-match accuracy. The shipped model is ~10 MB and runs fully on-device, no network.

4. Ternary classifiers for web elements worked

The next step had a concrete consumer: a browser-automation agent that scans a page and finds 50–500 interactive elements. I wanted tiny on-device classifiers to type those elements — is this a date picker? a login form? a search box? a consent banner? Each is a ~1.5M-parameter encoder that ternarizes to about 550 KB; a mesh of five is 2.7 MB total, classifying an element in under a millisecond on the Apple Neural Engine.

The training data is where the quality lives. It isn’t synthetic here — it’s harvested. A separate research crawler loads the Tranco top sites in headless Chrome and dumps each page’s accessibility tree and DOM; an extractor walks those trees and emits one row per interesting element, as a signature string built from class tokens, data-* attributes, and a few sample texts, with build-unique CSS-module hashes stripped so the signatures generalize across deploys. Rule-based labels on role and name (no LLM in the loop) reach F1 0.77–0.96.

The QAT-vs-rounding comparison came out as the literature predicts:

Classifier	fp32 F1	post-train ternary	QAT ternary
`is_date`	0.990	0.960	0.983
`is_login`	0.896	0.857	0.909
`is_search`	0.957	0.880	0.957

Rounding at the end costs 3–8 points of F1; training with the constraint recovers them, and on is_login the ternary model slightly exceeds full precision — the same effect Trained Ternary Quantization reported in 2017.

5. Training a base from scratch: the rung ladder instructive

Reproducing other people’s quantization is one thing; I wanted to know whether I could train a general ternary base from scratch, BitNet-style, on a budget. I built it as a ladder — each “rung” a larger model on more data, each one de-risking the next.

Rung	Params	Tokens	Perplexity	Cost	Purpose
1	50M	0.9B	133	~$15	does the pipeline converge at all?
2	99M	2.8B	44	~$26	fix the recipe, get real grammar
3	335M	7B	—	~$100 total	full-scale base, production target

Two practical notes. Initialization mattered more than in normal training — the usual init scale doesn’t suit a model that ternarizes around a threshold every forward pass, and reducing it by roughly 3× was the difference between divergence and stable training. Capability switched on abruptly with scale: on a structured task (instruction → JSON tool call), Rung 1 scored ~0%, Rung 2 ~50%, and Rung 3 reached 90% intent accuracy with 100% valid JSON, running in about 243 ms on an M3 MacBook Air’s neural engine.

The Rung-1 side experiment: can you LoRA-train something this small?

At Rung 1 I ran a separate small test that seeded everything later: can you LoRA-fine-tune a model this tiny and have it actually learn the task? Yes. On Rung 2, an adapter worth ~2.5% of the base raised grammar accuracy from 93.7% to 96.8%. More important than the number, this established the pattern I’d rely on from then on — one frozen ternary base, plus small task-specific LoRA adapters.