Haku Lab

Experiment

Slice adapters vs LoRA on a ternary model

Small on-device models are usually specialized with LoRA: a frozen base plus a low-rank side path per task. On a ternary base, that side path usually stays unmerged, because merging means requantizing. We tried a different unit: a ternary slice adapter. It retrains one slice of the model's feed-forward channels in place, at ternary precision, while every other weight stays frozen. Nothing is added to the graph, so the deployed model keeps the base model's shape.

We compared slice adapters with LoRA on conversation, app commands and tool calling, on four ternary models, including our 203M-parameter ternary mixture of experts and BitNet b1.58-large. On the M1 Neural Engine the slice adapter adds no decode time; a LoRA side path adds 1.7% to 41% depending on rank. Merging a LoRA into the ternary weights loses most of its gain. On our model and on BitNet b1.58-large, slice adapters match or beat LoRA of equal size on all three tasks. This post is the short version of a paper that is not public yet.

−0.2%
slice adapter decode time
vs base (M1 ANE)
+1.7–41.4%
LoRA side path decode time,
rank 8 to 480
1.835
conversation loss, slice adapter
vs 1.889 equal-bytes LoRA
11.75 MB
one slice adapter,
2-bit codes + scales
Scope. Four ternary models; the main claim holds on two of them (our 203M MoE and BitNet b1.58-large) and stops on the other two. All speed, memory and switch-time numbers are from one 8 GB M1 MacBook Pro through Core ML on the Neural Engine, apart from one labelled GPU control. Quality is scored in PyTorch on the training GPUs. Main arms on our model have 3 seeds; several larger-LoRA and BitNet points have 1 seed, marked below. Training data is machine-written. We did not test quantized LoRAs, DoRA or LoRA+, other chips, or energy.

The idea

Our base model is a small ternary mixture of experts: 12 layers, 4 experts per layer, every projection weight −1, 0 or +1 times a scale. Its experts are already separate units. A slice adapter is one expert retrained on a task — 1,920 of the 7,680 feed-forward channels in every layer — at ternary precision, with every other parameter frozen. We hash the frozen parameters before and after training; they are byte-identical.

At run time the whole request is routed to that expert. The deployed graph is the base graph with different codes, so the slice adapter costs exactly what the base costs. Stored as 2-bit codes with one fp16 scale per 128-weight block, one slice adapter is 11,750,400 bytes (11.75 MB).

LoRA is the obvious alternative. We applied it to the same expert (the w1, w3 and w2 projections in all 12 layers). At fp16, the equal-bytes LoRA is rank 64: 11.8 MB against the slice adapter’s 11.75 MB.

Method

TaskTrainTestMetric
Conversation (SODA-41k)41,000 rows12,186 rowsresponse loss, lower is better
App commands2,200 rows, 55 intents440 rows by an unseen writer; 1,090 rows of outside textintent exact; app accuracy
Tool calling (tc41k)41,580 prompts4,508 rows; BFCL v3 simple, 400 rowssemantic exact

LoRA ranks 8 to 480, same expert, same data, steps and seeds. The two methods want different learning rates, so each arm was swept until its best rate was inside the grid, and each is reported at its own best rate. Our first comparison used one shared rate, and it was misleading in both directions.

Results on our model

Conversation, valid loss, mean (min–max) over seeds:

ArmBytesSeedsValid lossM1 ANE decode vs base
Foundation, no adapter–13.413–
Slice adapter11.75 MB31.835 (1.834–1.836)−0.2%
LoRA r6411.8 MB31.889 (1.888–1.889)+3.0%
LoRA r12823.6 MB31.865 (1.864–1.865)+5.7%
LoRA r25647.2 MB11.845+16.3%
LoRA r48088.5 MB11.828+41.4%

At equal bytes the slice adapter beats r64 by 0.054; seed spreads are 0.002 or less. Only r480 scores lower, on one seed, at 8× the bytes and +41.4% decode time. On an earlier full-epoch run (one seed, one shared rate), r480 won outright, 1.421 vs 1.447.

Tool calling, test split semantic exact (n=4,508), 3 seeds:

ArmBytesTest split %BFCL simple /400
Slice adapter11.75 MB77.05 (74.62–80.06)67.3 (60–71)
LoRA r81.5 MB70.37 (68.83–71.74)12.0 (1–31)
LoRA r6411.8 MB75.22 (74.56–75.55)68.3 (65–71)
LoRA r12823.6 MB76.63 (73.62–78.59)70.3 (68–72)
LoRA r25647.2 MB76.59 (74.49–78.57)65.7 (63–69)

Seed ranges overlap, so the row-paired test carries this one: against r64 the slice adapter is significantly better in 2 of 3 same-seed pairs (exact McNemar, p < 0.05) and tied in the third. r128 and r256, at 2–4× the bytes, are within the slice adapter’s seed range. BFCL simple is a tie between the slice adapter and r64–r256.

App commands, in-distribution intent: a tie, 96.1 vs 95.5 for r64 (n=440, 3 seeds each). On 1,090 rows of realistic outside text the slice adapter leads on app accuracy, 77.4 vs 71.4 — but r64 at a different rate reached 74.8 (71.6–77.3), which overlaps.

What a side path costs on the Neural Engine

Decode time on the M1 Neural Engine, n=6,400 decode steps per arm:

ArmDecode median msvs basePackage MB
Base2.6150.0%36.5
Slice adapter2.611−0.2%36.5
LoRA r82.660+1.7%38.1
LoRA r642.695+3.0%48.4
LoRA r1282.763+5.7%60.2
LoRA r2563.041+16.3%83.8
LoRA r4803.698+41.4%125.1

Random adapter weights; timing depends on shapes, not values. Trained arms were scored on the ANE separately and stayed within ±3 rows of their GPU scores in 15 of 15 cells.

Any side path costs about 1.7%, even at rank 8: the graph gains 108 ANE operations whatever the rank. The cost grows slowly up to r128 and steeply after. Moving the side path to the GPU does not help: every arm decodes fastest on the ANE.

Why not merge the LoRA?

Merging would remove the side path. We merged 18 trained LoRAs into the ternary expert and requantized. Most LoRAs move almost no ternary codes — the per-weight change is 0.04–0.12 of a ternary step — so rounding erases them. In no arm did a merged LoRA match its unmerged self or beat the slice adapter. The best case went from 1.404 unmerged to 1.702 merged, against a slice adapter at 1.646 (and that slice adapter was a weaker, low-learning-rate one).

Is this just our model?

Our base was built so one expert can be trained alone. To check that this design is not the whole story, we repeated the comparison on BitNet b1.58-large, a dense 729M ternary model. There a slice adapter is 400 of the 4,096 feed-forward channels in every layer — 44,236,800 weights, the same count as our expert — retrained in place.

BitNet b1.58-large armBytesConversation loss (seeds)Tool call % (n=600, 1 seed)
Slice adapter11 MB1.2705 (3)66.0
LoRA r86.5 MB1.2923 (1)60.2
LoRA r1613 MB1.2856 (3)60.3
LoRA r6452 MB1.2670 (1)62.7

The slice adapter beats the equal-bytes LoRA (r16) on conversation and tool calling, and adds no decode time on the M1 ANE (−0.014 ms, against +0.216 ms for r16). A LoRA at 4.7× the bytes (r64) matches it on conversation, on one seed. App commands tie at the ceiling (97.8 vs 97.8).

On two smaller dense ternary models of ours (99M and about 350M, with no activation quantization), the result is weaker. The slice adapter leads the equal-bytes LoRA on conversation by a small margin, on one LoRA seed, but a LoRA at 1.6–2.5× its bytes beats it. On the 99M model the equal-bytes LoRA also beats it on app commands, 89.5 vs 83.4 (one seed each; the slice adapter ran at one learning rate there). That is where the claim stops for now.

Limits

  • Format brittleness. On tool-schema renderings never seen in training, the slice adapter breaks its output format more often: 49% malformed on the worst rendering (59 of 120 rows over 3 seeds) vs 12% for the LoRAs (72 of 600).
  • Quantized LoRAs. Stored in 8 bits, the equal-bytes LoRA becomes r128, and the quality lead holds on conversation only. We did not train or time quantized LoRAs.
  • Codes-only models. On a model released without latent weights, results were inconclusive.
  • Switching. A slice adapter is not yet its own file: on the ANE it is compiled into a 36.5 MB whole-model package. Switching takes 113.8 ms (median, n=12) while that package stays at a fixed path; a new path recompiles for about 2.2 s.

Where the slice adapter wins, it wins on bytes and device cost rather than on rank. A rank-64 truncation of a slice adapter’s change keeps 99% of its gain on app commands and 81% on conversation, so a LoRA could hold most of it. It needs more bytes and a side path to do so.

Get new experiments

Occasional notes when we publish. No spam.