Parallel decoding of audio tokens in a text-to-speech model

A neural TTS model emits audio tokens, not a waveform. MOSS-TTS-Nano produces, per 80 ms frame, 16 codes from a residual codebook stack: codebook 0 is coarse and carries the linguistic content; codebooks 1–15 are finer residuals carrying timbre and texture. The model normally decodes the 16 sequentially — sixteen forward passes per frame, which is most of the decode cost. We replaced this with a single parallel pass (a masked, non-autoregressive decode in the SoundStorm / VALL-E family) that predicts all 16 codes at once, and measured speed, transcription accuracy, and codebook agreement.

Result in one line. ~2× faster decode; transcribes correctly; the fine codebooks degrade, which is audible but does not show up in transcription. Reported because the automated metric and the listening test disagree.

Speed

Given the same upstream hidden state, the parallel decoder is byte-identical to the sequential one (100% token-exact parity). Measured on-device (Apple Silicon):

per frame	sequential	parallel	ratio
local-stage forwards	17	2	8.5×
local-stage time	18.3 ms	4.2 ms	4.4×
total decode time	27.3 ms	13.5 ms	2.0×

Transcription

We synthesized unseen sentences with the parallel decoder and ran the audio through a speech recognizer, comparing the transcript to the input text. Word-error-rate was effectively zero — the transcripts matched the input verbatim, the same as the original model. Example: “The lighthouse keeper watched the storm roll across the harbor at dawn.” transcribed back word-for-word.

Codebook agreement

On unseen text, agreement between the parallel and sequential decoders, measured per codebook:

codebook	role	agreement
0	coarse — linguistic content	~61%
1–15	fine — timbre, texture	~2–9%

Transcription depends on codebook 0, which mostly transfers; perceived voice quality depends on codebooks 1–15, which do not.

Played back, the parallel speech is intelligible but degraded — the words are correct, the voice quality is not. This matches the codebook numbers and is not visible in the transcript.

Method and interpretation

Residual codebooks are hierarchical: codebook k refines codebooks 0…k−1. The sequential decoder predicts each codebook conditioned on the codes already produced. The parallel decoder predicts all 16 from the frame’s hidden state alone, so the fine codebooks are predicted without the coarse codes they refine. We also tried a coarse-to-fine reveal schedule (a few rounds instead of one pass); per-codebook agreement on the fine codebooks did not improve (8.1% vs 8.6%), so the limit is not the decode schedule.

Limitations

Trained on a small dataset (350 sentences). The coarse codebook generalizes to unseen text; the fine codebooks do not, at this scale and conditioning. Transcription accuracy is not a sufficient quality metric here, because it reads only the coarse codebook. Closing the quality gap would require strict coarse-to-fine conditioning (predict each codebook given the ones below it, trained that way) and more training data — not attempted here.