Running a ternary 27B on a single A10

A 27B model whose weights are all −1, 0 or +1 — what it takes to run, and how fast it goes.

Prism ML publish Bonsai as a size series — 1.7B, 4B, 8B, 27B — one ternary recipe scaled up, every weight in {−1, 0, +1} at a true 1.71 bits. The point of ternary is where each model lands: the 1.7B is 0.46 GB and runs on a phone, and the top of the range, a 27B, packs to a 7.17 GB file that fits a 16 GB laptop — a size class that in FP16 is a ~51 GB datacentre model. This is the deployment note for that 27B: what it took to build and run, and how fast it went on a single A10 and, from the model card, on an M-series laptop. Whether it is good — whether it holds together as an agent — is the other experiment.

The size series

It is one recipe at four scales. The footprints below are the actual GGUF files — the whole point is which device each one lands on:

Model	File	Fits
Ternary Bonsai 1.7B	0.46 GB	a phone
Ternary Bonsai 4B	1.07 GB	a phone
Ternary Bonsai 8B	2.18 GB	any laptop
Ternary Bonsai 27B	7.17 GB	a 16 GB laptop

The weights are ternary end-to-end — embeddings, attention projections, MLP projections and the LM head — stored as −1, 0, +1 in 2-bit slots with one FP16 scale per group of 128, which is where the 1.71 bits per weight comes from (against 16 for FP16). The smaller sizes have been in the app for a while; the rest of this note is about getting the 27B, the new top of the range, running.

Which side of the line

For the 27B specifically, the compression ratio matters less than which side of a hardware line the result lands on:

27B build	Weights (disk)	Peak RAM, 4K ctx	Peak RAM, 100K ctx
bf16 (dense)	51.25 GB	52.6 GB	59.3 GB
Q4_K_XL (“4-bit”)	17.6 GB	19.2 GB	25.6 GB
Ternary (Q2_0), 4-bit KV	7.15 GB	8.4 GB	~10.1 GB

Peak RAM (weights + KV cache + runtime buffers) held resident during inference — not disk. The disk file is the 7.17 GB weights; everything above that is working memory. The ternary row uses a 4-bit KV cache; with an uncompressed FP16 cache the 100K peak is 14.7 GB instead.

A dense 27B needs a workstation. The conventional 4-bit build is a 32 GB machine at best, and still runs out at long context — 25.6 GB before you have loaded anything interesting. The ternary build sits inside a laptop: 8.4 GB at short context, and a 100K-token context in ~10.1 GB once the KV cache is 4-bit — both comfortable on a 16 GB machine with the operating system still resident. The cache stays cheap for two reasons: the backbone is hybrid-attention, so only 16 of 64 layers grow a full-attention cache, and the cache quantises well — ~65 KB per token at FP16 drops to ~16 KB at 4-bit.

That cache line is also the part still improving. Four-bit KV is standard and already in the runtime we used (with a --kv-mean-center option to sharpen it); newer methods go lower at the same quality — Google’s TurboQuant (ICLR 2026), now landing in llama.cpp as a ~3-bit cache type, reconstructs the attention state with a provable distortion bound. The direction of travel is that the 16 GB machine gets more context headroom over time, not less — the memory budget here is a floor that keeps dropping, not a ceiling to fear.

Building for Ampere

The packed weights need kernels that consume them directly, which live in a fork of llama.cpp. Every CUDA number Prism publish is from an H100, so the first question was whether the kernels build for an older, smaller GPU at all. They do — -DCMAKE_CUDA_ARCHITECTURES=86, clean compile, no patches. We ran on a single A10 (24 GB, 600 GB/s) on Lambda at $1.29/hr, and measured with llama-bench:

Machine	Prefill (pp512)	Decode (tg128)
H100 — published	2,596 tok/s	98.0 tok/s
A10 — measured here	888.9 ± 8.9	40.3 ± 0.5
M5 Max — published	830 tok/s	44.0 tok/s
M5 Pro — published	393 tok/s	26.2 tok/s
M4 Pro — published	125 tok/s	18.0 tok/s

The A10 row is ours; the rest are the card’s.

The laptop beats the server

Put the rows next to each other and the surprising line falls out: an M5 Max decodes this model at 44 tok/s and the datacentre A10 we rented did 40.3. The laptop is not a degraded port of the server deployment — it is faster than the server we paid for. Decode at batch 1 is memory-bandwidth-bound, not compute-bound, and Apple’s unified memory has the bandwidth.

The prefill column tells the same story from the other end. An A10 has roughly an order of magnitude less FP16 compute than an H100, yet it is only 2.9× behind on prefill — because the batch-1 low-bit path is not compute-saturating an H100 in the first place. Most of the per-token cost at batch 1 is kernel-launch and synchronisation latency across 64 layers, and that does not shrink on a bigger GPU. The card says as much about decode, where the ternary and 1-bit builds converge (98 vs 104.8 tok/s) despite moving 1.9× different bytes. A mid-range card gives up far less than its spec sheet predicts, and a laptop with enough bandwidth is competitive with a rented GPU.

Possibilities

A 27B that peaks at 8.4 GB and decodes at 26–44 tok/s on an M-series laptop is the first model of its capability class that could run on the machine it is meant to serve, rather than behind an API. That opens a few things that were not on the table before:

An agent that runs where it acts. A model driving a browser or a Mac app can reason on the same machine, so nothing about the task — the page, the files, the messages — has to leave the device to be understood. No round-trip, no per-token bill, and it works with the network off.
The series as a deployment ladder. The same recipe runs from 0.46 GB on a phone to 7.17 GB on a laptop. A task can be sent to the smallest size that clears it and escalated only when it needs the depth — a size dial rather than a single model.
A long context that stays local. With a 4-bit KV cache the hybrid-attention backbone holds a 100K-token context in ~10.1 GB of RAM, so a whole document or a long session fits in a 16 GB laptop without a server in the loop.

The honest limits are in the same numbers. Decode at 26–44 tok/s is usable but not instant, and prefill on a long prompt is the real latency — a big context has to be read before the first token. This is a serving profile for a thoughtful on-device agent, not a low-latency chat box. Whether it earns that place — whether it reasons like a 27B once it is running — is the next experiment: Does the 27B Bonsai actually deliver 27B?

Links

Ternary Bonsai 27B — huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
low-bit kernels — github.com/PrismML-Eng/llama.cpp