An experiment to cut down TTS latency

The bottleneck

Per 80 ms frame, the model emits 16 codes. Code 0 (the “coarse” code) is produced by the main model; codes 1–15 are produced by a small second model — the code predictor — that runs autoregressively over the codebooks: it predicts code k given the frame’s hidden state and codes 1…k−1. Fifteen sequential forward passes per frame. On an A100, that fine stage is ~24 ms of a ~33 ms frame — the coarse model is the other ~9 ms. Speed up the fine stage and you speed up the whole thing.

First attempt — decode all of them at once

Predict all the fine codes in one parallel pass instead of sequentially (the masked, non-autoregressive idea from the SoundStorm / VALL-E family). We tried this first on a small TTS model (MOSS-TTS-Nano), then here with a head trained from scratch. It is faster, and the result transcribes correctly. But played back it is intelligible and rough: the coarse code transfers, the fine codebooks do not. The reason is structural — a parallel pass predicts the fine codes without the lower codes they refine. The transcription metric and the listening test disagreed, and the listening test was right.

The words are there; the voice is buzzy and thin. The rest of the article is about closing that gap — compare with the final one-pass audio below.

Could we keep it exact instead?

Before accepting any quality loss, we checked whether the parallel decode could be made lossless, with speculative decoding: a cheap parallel drafter proposes all the codes, the real autoregressive model verifies them in one pass and keeps the longest prefix it agrees with. The output is then identical to the slow model whenever the drafter guesses right. On constrained text this can pay off (we measured acceptance on a small probe). On acoustic fine-codes it does not. The codes are so sequentially dependent that a one-shot parallel draft agrees with the autoregressive model for only ~1.5 codes before diverging, and the drafter’s forward pass costs about as much as the verifier’s — so the overhead outweighs the saving. A lossless fixed-point (Jacobi) version needed ~12 rounds to converge, slower than the 15 sequential passes it replaces. So lossless does not transfer to acoustic fine-codes. To get the speed, we had to accept some loss and keep it small.

The pivot — adapt the existing model

The first attempt trained a parallel head from scratch. A better approach: take the real code predictor, freeze it, and add a small low-rank adapter (a LoRA) that re-purposes it to predict all the fine codes in one bidirectional pass. A frozen, already-trained model carries representations a from-scratch head would have to relearn, so the adapter should reach higher quality at the same one-pass speed. It did: on the same data and held-out set, the adapter on the frozen model did better than the from-scratch head, and generalized where the small head had memorized.

A data bug that hid the result

For a while the adapter looked weak — the data was misaligned. The training pairs were (frame hidden state → codes), and the hidden state we had captured differed from the one the code predictor actually consumes. The check: feed the real model the true codes and it should reproduce its own output almost exactly. It did, 95.9%, but only with the correctly captured hidden state; with the one in our dataset, 2.4%. A simple gate — does teacher-forcing reproduce the model — caught it. Fixing the capture lifted first-codebook agreement from 22% to 74%. The earlier “the fine codes are unlearnable” conclusion was mostly this bug. The lesson: gate your data against a check the model itself can answer.

Scaling, and matching the original

With aligned data, the remaining levers were adapter size and data: rank 48 → 128, and 1.5k → 24k sentences. Quality climbed and then flattened — diminishing returns on exact-code agreement, but the word-error-rate reached the original’s: a speech recognizer transcribes the one-pass output and the real model the same.

Word-error-rate is the recognizer’s transcript vs the input text; the original isn’t 0 because the recognizer and codec aren’t perfect. The one-pass adapter matches it.

Listen — the final one-pass

Held-out sentences (the model never saw them in training). For each: the original autoregressive decode, then the final one-pass adapter. The words are identical; listen for the fine texture — and compare against the rough first attempt above.

Where it still falls short

It is not bit-exact, and you can hear it: a faint “unpolished” texture. We measured exactly where it comes from, over 60,000 held-out frames. Agreement with the original decays smoothly down the codebook stack — but the more telling number is whether the true code is even near the top of the adapter’s prediction:

Mean rank is out of 2048 codes. For the low codebooks the true code sits at the top of the distribution; for the high ones it is buried — the prediction there is nearly flat.

This splits the residual cleanly. The low codebooks are recoverable — the right code is in the top few, so it carries real, learnable detail. The high codebooks are genuinely high-entropy: from a single pass, the finest residual is close to unpredictable (the original only “knows” it because it conditions on the realized codes below it). And the errors are diffuse, not glitchy — almost every frame has 12–13 of its 15 codebooks off, which is why the artifact is a uniform texture rather than occasional pops, and why simply muting the high codebooks (we tried) doesn’t clean it up. There is no systematic confusion to patch either; the errors are spread, not a fixable bias.

This is the same structural limit as the first attempt, much smaller: a single parallel pass predicts each code without the realized codes around it. Two passes would help, but two passes cost about what the original fifteen do — at which point you would run the original and get an exact result. So there are two useful settings: one pass at near-original quality, or the original for an exact result. Little in between is worth the cost.

Speed

The one-pass adapter is a full pass through the (small) frozen model, so it is heavier than a from-scratch head — that cost is what recovers the quality. The coarse stage (~9 ms) is unchanged and sets the floor on the end-to-end ratio.

Setup

Limitations

One model; one language’s worth of held-out text; quality judged by transcription word-error-rate and listening, not a formal MOS study. The remaining texture artifact is real and unfixed within the one-pass budget. Timings are from a cloud A100, not a Mac — the ratios are the point, not the absolute milliseconds. This is a speed/quality trade-off, not a new state of the art.