An experiment to format dictation in under 100 milliseconds

A 60M-parameter T5, distilled from a larger model, that cleans up raw speech on-device — split across the Neural Engine and the GPU.

On-device dictation gives you accurate text, but it is raw: run-on sentences, no paragraph breaks, "um" and "you know" left in, a spoken list that should be bullets. Turning that into clean, formatted text is a small, well-scoped rewriting task. The obvious way to do it is to hand the transcript to a general language model — we first tried a small one, a 1.7B, which did the job well but took about 3.7 seconds per utterance. That is too slow to feel instant, and far more model than the task needs, so we built a model specifically for it.

It is a 60M-parameter T5, distilled from a larger model, with a serving path that matches its shape: the encoder runs on the Apple Neural Engine, the decoder on the GPU. It formats a typical utterance in under 100 milliseconds, entirely on the Mac, with the encoder and decoder each numerically matching the original model.

The problem

The speech recognizer is already good; what it returns is not. A spoken paragraph comes back as one long line, fillers intact, no structure — legible, but not something you would paste into an email as-is. Rewriting it for form — without touching the content, and deciding whether it is prose or a list — is a narrow, almost-deterministic task, which is why a 1.7B is the wrong tool for it: a model two orders of magnitude smaller should learn it. So we distilled it into a fine-tuned T5-small (60M), trained on generated pairs of messy dictation and its clean formatting across the cases that matter — email, notes, to-do lists, messages.

Two halves, two chips

The interesting part is serving it. T5 is an encoder–decoder model, and the two halves want different hardware. The encoder reads the whole input in one parallel pass at a fixed shape — what the Neural Engine is built for. The decoder generates one token at a time against a growing key/value cache — a sequential loop that belongs on the GPU.

So we split it. The encoder is emitted as a CoreML program (hand-built in MIL, fixed at 128 tokens) and runs on the ANE. The decoder is hand-written Metal — T5’s no-scale attention, the relative-position bias added straight to the scores, cross-attention into the encoder output, a ReLU feed-forward — on top of the GPU decode loop and KV cache we already had. Getting fp16 numerics right is the whole game here, so each half was held to the original model: the encoder matches to a cosine similarity of 1.00000, and the decoder picks the same token at every position.

Results

End to end it formats a typical utterance in 80–115 ms on an M-series Mac. Try it — this calls the model running live on a Mac, not a cloud API:

It adapts its layout to the content:

you saidthe action items from the meeting are update the roadmap review the budget and schedule the client call

it wroteThe action items from the meeting are:

Update the roadmap.
Review the budget.
Schedule the client call.

you saidthe design looks good but um the colors are a bit off maybe we can try something softer you know

it wrote

The design looks good, but the colors are a bit off. Maybe we can try something softer.

Live outputs from the 60M model. The fillers are removed, the prose is split into sentences, and the list becomes bullets.

stage	hardware	time
encoder	Neural Engine	4–30 ms
decoder	GPU (Metal)	60–150 ms
total	—	~80–115 ms
1.7B baseline	GPU	~3,700 ms

The weak spot is list detection. The model bullets reliably when the speech carries an explicit cue — “the action items are…”, “a to-do list…”, “first… then…” — but on a bare run-on it often keeps everything as one sentence, and it under-punctuates the items it does bullet. That is the part of the synthetic set that needs the most coverage, and it is the obvious next round of training.

How Haku uses it

It runs in Haku Dictate, the on-device dictation app. Formatting is surface-aware: in the apps where structured text belongs — notes, mail, documents, the browser — the formatted version is written; in a terminal or a code editor the raw transcript is left untouched, because reformatting a command would break it. The transcript never leaves the device, and now neither does the cleanup.

Links

T5 — github.com/google-research/text-to-text-transfer-transformer