Haku Lab

Experiment

An experiment to run LiquidAI's DSpark drafter on the Neural Engine

The Neural Engine handles prefill well — our engines already use it for that. Autoregressive decode is the part that resists it: mutable KV state, dynamic lengths, one token per call. LiquidAI's DSpark draft model — the speculative drafter that speeds up LFM2.5-1.2B — sits on the decode path but is shaped like prefill: it proposes a fixed 9-token block once per round, attends bidirectionally, gets its context KV injected from the target's hidden states rather than carrying its own, and is precision-insensitive because the target verifies every draft exactly. We moved it onto the ANE while our Metal engine keeps verification. Output is byte-identical to plain decoding; the drafts got 2× faster; and on a machine that is already saturated, the whole pipeline got 39% faster.

11.3 ms
drafter on ANE
vs ~23 ms on GPU
+39%
end-to-end on a
saturated 8 GB Mac
+4%
end-to-end on an
idle M1
0
tokens changed
(verification is exact)
Scope. One drafter (DSpark for LFM2.5-1.2B) and two machines: an M1 MacBook Pro for the drafter timings and the idle end-to-end runs, and an 8 GB MacBook Air under other workloads for the saturated run. Three workload types end-to-end (structured output, step-by-step, prose). The number of prompts and runs behind each figure is not given here. Output was checked byte-for-byte against plain decoding.

Why the drafter fits

DSpark is DFlash-style speculation. Per round, the drafter sees the last committed token plus eight mask tokens and proposes nine tokens in one forward pass; the 1.2B target then checks all nine in one batched pass and keeps the longest correct prefix. Two properties make the drafter ANE-shaped where the rest of an LLM is not.

First, its context cache is not state. The drafter never runs over the sequence — for every committed token, the target’s hidden states at five tap layers are projected into draft K/V and written into the drafter’s cache. So the cache can be handed to Core ML as a plain fixed-size input tensor each call, sidestepping the stateful-model path that the ANE rejected in our earlier full-decode attempts. Committed rows never change, so only the few new rows are copied per round.

Second, precision does not matter. The verifier is exact: if the ANE’s fp16 rounds a draft differently, the worst case is one fewer accepted token, never a different output. In our tests it did not even cost that — the ANE proposed the same drafts as the Metal drafter on most prompts, and the emitted text was byte-identical everywhere.

Method

The drafter’s five attention layers are exported as a fixed-shape fp16 mlprogram (9-token block, 1024-token context window, RoPE tables and attention mask as host inputs), built directly with a MIL builder — no torch in the loop. The Metal engine keeps everything else: verification, KV injection, the output head, and the Markov correction chain that sequences the nine proposals. Per round the two processors alternate: the ANE drafts, the GPU verifies.

Results

On an M1 MacBook Pro, one drafter forward including input transfer:

Enginems / draft block
Neural Engine11.3
GPU (same engine, Metal)~23
CPU17.6

End-to-end, with everything else identical and output byte-identical, the ANE drafter wins on every workload — 67.0 vs 58.8 tok/s on structured output (+14%), 45.3 vs 43.0 on step-by-step, 32.1 vs 31.6 on prose. The gain tracks how much of each round is spent drafting.

The larger result came from a loaded machine: on an 8 GB MacBook Air deep in swap under other workloads, the ANE configuration ran 24.9 vs 17.9 tok/s — +39%. The ANE has its own execution resources, and 460 MB of drafter weights leave the GPU’s working set. Since on-device inference mostly happens on machines that are busy doing other things, the saturated case is the one that matters.

The stall that wasn’t the ANE

The first integration was slower than Metal: the same call that benchmarked at 11.5 ms back-to-back took 24.6 ms inside the decode loop. Probes, in order: a keep-warm heartbeat model pinging the ANE every 10 ms — no change. Mutating all 10 MB of inputs between back-to-back calls — no change. A plain 15 ms sleep between calls — latency doubled, no GPU involved.

Between callsms / call
nothing (back-to-back)11.5
15 ms sleep22.9
100 ms sleep23.4
15 ms busy-wait12.8
100 ms busy-wait12.3

Same model, same inputs, idle M1. The Core ML CPU path shows the same effect, harder: 17 → 51 ms after a 100 ms gap.

When the calling thread blocks — on a GPU wait, on anything — macOS downclocks the core, and the next prediction pays ~13 ms of CPU-side ramp before the ANE sees work. One thread doing light floating-point busy-work while the pipeline runs holds the clocks up; the in-loop call returned to 11.4 ms. This applies to any latency-sensitive pipeline that alternates between Metal and Core ML.

What didn’t work

Quantizing the drafter backbone to int8 made the ANE slower — 8 ms worse than fp16, the blockwise dequant dominating at these layer sizes. int8 paid off only on the 65536×2048 output head. And fusing the full sampling chain into the ANE graph — nine unrolled gather→argmax steps — produced byte-identical tokens but lost 25 ms per call to ANE↔CPU engine transitions. The chain stays on the GPU, where it costs 4 ms.

Get new experiments

Occasional notes when we publish. No spam.