Haku Lab

Experiment

Fixing small models by editing their weights: 5 of 7 failures

An MIT CSAIL post, "Sparks of Weight-Space Programming" (2 Nov 2026), gave a strong LLM a small model's weights, a Python sandbox and a budget of forward passes, and no training. It found that Qwen3.5-0.8B loops until it hits the token limit, edited 64 MLP output columns in each of two layers, and MATH-500 went from 7.9% to 48.1% (with 50 forward passes). We asked a practical question: which failures can a direct edit fix, and can you tell before you try? Over 7 failures in 6 of our own small models, 5 were fixed on held-out data, and the split was clean. Editing works when the failure is one choice at one token: a tag, a digit, when to stop. It failed every time the model had to copy or write the right content. The paper's own loop fix failed on our chat model, for a reason that turned out to be the most useful result here.

5 of 7
failures fixed
on held-out data
150 → 63
looping answers
of 265 (chat model)
1,280
numbers changed
for the loop fix
0 of 2
copying failures
moved
Scope. One editor (Claude, reading weights and activations through Python scripts), one seed and one run per model. 7 distinct failures on 6 models, 9 edit attempts in all. Every edit was scored once on held-out data, beside a random edit of the same size. None of these edits has shipped: a benchmark gain is a reason to try an edit, not proof that it works for users. First published as an article on X on 3 Nov 2026.

The question

The MIT authors say editing “could remove an obstacle that hid an ability, but it could not add an ability the model lacked.” Their rules banned training and banned edits to the embedding or the output head.

Our hypothesis: an edit can fix a failure only when the model already has what it needs and makes one wrong choice. When the missing piece is a skill, such as copying the right words, no small edit will help.

Method

Locate. On practice data, compare the model’s insides where it fails with where it succeeds. Look at the token where the decision is made, and rank the neurons that push toward the right or the wrong answer.

Edit. Change only those weights. Most of our test models are ternary: every weight is −s, 0 or +s. So the edit had to keep that form. One way is to copy a helpful neuron into an unused one, which doubles its effect using values already present. Another is to change the pattern of −1, 0 and +1 while keeping the tensor’s scale fixed.

Dose. Try a few strengths on practice data only, and lock the choice before looking at test data.

Random control. Make an edit of the same size to random neurons. If it moves the score too, the “fix” is noise.

Held-out, once. Score the locked edit on data it never saw. Count rows fixed and rows broken, and compute beforehand the smallest effect the test can detect.

Read the outputs. Benchmarks miss things. For chat, we read the answers side by side.

Test cases

Small task models of 13.5M to 88M weights, each with one known failure and a held-out set.

ModelThe failureKind
Date parserInvents an hour for “haircut on friday”one choice: “no time” vs an hour
Date parserReads “the day after tomorrow” as +1 dayone choice: a digit
Reminder taggerSwallows “on friday” into the titleone choice: a tag
Text formatterAdds too many paragraph breaksa threshold, spread across the network
Chat model (MoE expert)Answers loop until the length capone choice: when to stop
Invoice extractor (MoE expert)Loops on the vendor fieldcopies the wrong span
Reminder writerWrites stock words instead of the titlecopying gap

What happened

Held-out numbers, scored once.

ModelEdit · weights changedBefore → afterFixed / brokeRandom editWorked?
Date parser: “no time”copy 5 neurons ×6 · 18,952 of 14.5M (0.13%)invented hours 152 → 42 of 30090 / 0149 of 300yes
Date parser: “+2”copy 10 neurons ×6 · 37,913 (0.26%)27 → 81 of 20054 / 226 of 200yes, with a side effect
Reminder taggercopy 10 neurons ×4 · 23,809 of 14.95M (0.16%)251 → 280 of 32634 / 5250 of 326yes
Text formatterlower 2 output biases · 2 numbersparagraph F1 .671 → .7571,156 / 309 words±0.01 F1threshold only
Chat modelshift the stop token’s row · 1,280 numbersloops 150 → 63 of 26588 / 1155 of 265yes, shorter answers
Chat model, the paper’s methodremove the “repeat” direction · 54k–970k entriesdropped on practice data––no
Invoice extractorzero 5 neurons · 6,078 (0.014%)755 → 753 of 1,60024 / 26753 of 1,600no
Reminder writercopy 10 neurons ×4 · 22,717 (0.15%)169 → 167 of 3261 / 3169 of 326no

Formatter: n=71,183 words over 509 documents; list F1 also rose, .564 → .620. Reminder writer: this test could only detect a change of 36 rows or more. Every edit on a ternary model kept its tensors ternary.

Two more things showed up. The “+2” fix also turned 10 of 142 synthetic “N minutes ago” rows into “−2”, a case the practice set barely covered. And copying neurons was cleaner than any threshold change: it stayed inside the model’s own values and broke few rows.

The loop story

Loops were the paper’s headline, so we expected the chat model to be the easy case. It wasn’t, and why it wasn’t is the most useful result here.

The paper’s edit failed. We found the “repeat” direction in the chat model’s MLP outputs and removed it from the output columns, as the paper did. On practice data it lowered good tokens as much as loop tokens, so it damaged normal text without breaking the loops. The reason: inside a loop the model is near-certain. The repeated token has median probability 0.996, and stopping has 0.0000 (n=7,380 positions). There is no decision left to tip.

The real error comes earlier. At the token just before an answer starts repeating, the chance of stopping is 0.0000013 (n=155). At real endings it is 0.85 (n=95). The model runs past its own end and then falls into repetition. That makes the loop a single missed choice, the same kind as the date and tag failures.

The cheap test agreed. Pushing the stop token up a little at generation time cut practice loops from 155 to 73 of 276, and the instruction-following score barely moved (37 → 36 strict passes).

What worked as a weight edit: shifting the stop token’s row in the embedding along the model’s average hidden direction. It changes 1,280 numbers and acts like a fixed push toward stopping. The paper’s rules would have banned this edit. The version that stays inside their rules, copying the neurons that signal “stop” inside the expert, helped only partly (150 → 120 of 265).

IFEval held-out, 265 promptsLoopsHit length capStrict passEnds mid-sentence
original150130437
stop shift, moderate63193636
stop shift, strong2653889
random shift, same size155133436
copy “stop” neurons ×3120984413

The model is a ternary MoE chat expert (88M) on our 8K foundation, greedy decoding, up to 1,024 new tokens. Smallest detectable change: 32 rows for loops, 22 for strict passes. The pre-registered rule picked the strong dose; the moderate dose was declared as a second arm before any test scoring.

On 100 real user prompts (WildChat), loops fell from 62 to 31 with the moderate shift. A random shift of the same size gave 64.

Read as a user would. The edit keeps each answer’s opening and stops it at a paragraph end, before the repeat. It does not make the content better. The strong dose cuts answers mid-thought, and the instruction score did not notice; counting mid-sentence endings and reading the answers did. That is why the moderate dose, not the rule’s pick, is the usable one.

“What is an inverter”, original (251 tokens):

An inverter is a type of inverter that is used to convert a circuit from one mode to another. It is a type of inverter that is used to convert a circuit from one mode to another. Inverters are used to convert a circuit from one mode to another, and they are often used in combination with other inverters to achieve the desired output.

Inverters are typically used in a variety of applications, including:

1. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems.
2. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems.
3. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems.
4. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems.

Overall, inverters are a powerful tool for converting a circuit from one mode to another, and they are widely used in various applications.

Moderate stop shift (79 tokens):

An inverter is a type of inverter that is used to convert a circuit from one mode to another. It is a type of inverter that is used to convert a circuit from one mode to another. Inverters are used to convert a circuit from one mode to another, and they are often used in combination with other inverters to achieve the desired output.

A math prompt from IFEval: “11 results have an average of 51. The first five have an average of 49, and the last nine have an average of 52. What was the sixth result?” The original loops to the 1,024-token cap:

The sixth result is:

11 results have an average of 51.

The last nine have an average of 52.

So, the sixth result is:

11 results have an average of 51.

Now, we need to add the results of the last nine to the total number of results:

11 results have an average of 51.

The sixth result is:

11 results have an average of 52.

Adding the results of the last nine to the total number of results:

11 results have an average of 52.

Therefore, the sixth result is:

11 results have an average of 52.

So, the sixth result is:

11 results have an average of 52.

The sixth result is:

11 results have an average of 52.

[… these last two lines repeat until the cap]

Moderate stop shift (6 tokens): The sixth result is: — strong (4 tokens): The sixth result is

This is the paper’s loop failure on a math problem. The paper’s loops hid answers the model could reach. Here nothing was hidden: with the loop gone, the answer stops after “The sixth result is”. The correct answer is 152.

The rule, and the test to run first

Editable when the failure is one choice at one token, and the right token is one the model already produces elsewhere.

Not editable when the model writes or copies the wrong content over many tokens. Small edits can’t add a missing skill.

The test (minutes, no editing): on practice data, push the right token’s score up a little at generation time. If that alone fixes the failure, a weight edit can do it too. If there is no single token to push, don’t try.

One choice at one tokenContent over many tokens
Casesdate “no time”, date “+2”, tagger, formatter, chat stopinvoice span, reminder title
Push-the-token testfixes it (chat: loops 155 → 73 of 276)no single token to push (invoice: the “close” choice explains 19 of 1,000 loops)
Edit result5 of 5 fixed0 of 2

Which edit to use, once it passes: if a few neurons carry the decision and fire too weakly on the failing rows, copy them into little-used neurons of the same layer; that keeps a ternary tensor ternary. If failing and passing cases look the same inside the network (formatter: the largest neuron gap was 0.094), shift a threshold instead.

A side question: can an edit make a model ternary?

The same day we asked whether direct edits, again with no gradients and no training, could turn a dense model into a ternary one. On SmolLM2-360M-Instruct, the best pure-ternary edit closes 80.0% of the gap between the dense model and plain rounding, measured in log-perplexity on WikiText-2 (n=148 chunks): perplexity 128, against 12.7 dense and 1.3M for plain rounding. Keeping 2 of 32 layers dense closes 85.6% (perplexity 67). None of the 21 ternary versions answered any of 5 chat prompts correctly (0 of 5 each). So: yes, the weights can be made ternary by editing alone, but not into a usable chat model.

Limits

One editor and one run per model. The editor wrote some of the practice and test probes. The instruction-following set, the real user prompts and one handwritten reminder test set were written by others, which makes those the strongest held-out evidence.

The instruction score fell from 43 to 36 with the moderate stop shift (4 won, 11 lost). This test could only detect a change of 22 or more, so a real loss of that size can’t be ruled out.

The stop shift changes a shared tensor (the tied embedding), so in a model with several experts it affects all of them. It was not tested on other routes. The ternary-only neuron copy stays inside the expert.

None of these edits has been exported to Core ML or driven in an app.

Get new experiments

Occasional notes when we publish. No spam.