Experiment
Fixing small models by editing their weights: 5 of 7 failures
An MIT CSAIL post, "Sparks of Weight-Space Programming" (2 Nov 2026), gave a strong LLM a small model's weights, a Python sandbox and a budget of forward passes, and no training. It found that Qwen3.5-0.8B loops until it hits the token limit, edited 64 MLP output columns in each of two layers, and MATH-500 went from 7.9% to 48.1% (with 50 forward passes). We asked a practical question: which failures can a direct edit fix, and can you tell before you try? Over 7 failures in 6 of our own small models, 5 were fixed on held-out data, and the split was clean. Editing works when the failure is one choice at one token: a tag, a digit, when to stop. It failed every time the model had to copy or write the right content. The paper's own loop fix failed on our chat model, for a reason that turned out to be the most useful result here.
on held-out data
of 265 (chat model)
for the loop fix
moved
The question
The MIT authors say editing “could remove an obstacle that hid an ability, but it could not add an ability the model lacked.” Their rules banned training and banned edits to the embedding or the output head.
Our hypothesis: an edit can fix a failure only when the model already has what it needs and makes one wrong choice. When the missing piece is a skill, such as copying the right words, no small edit will help.
Method
Locate. On practice data, compare the model’s insides where it fails with where it succeeds. Look at the token where the decision is made, and rank the neurons that push toward the right or the wrong answer.
Edit. Change only those weights. Most of our test models are ternary: every weight is −s, 0 or +s. So the edit had to keep that form. One way is to copy a helpful neuron into an unused one, which doubles its effect using values already present. Another is to change the pattern of −1, 0 and +1 while keeping the tensor’s scale fixed.
Dose. Try a few strengths on practice data only, and lock the choice before looking at test data.
Random control. Make an edit of the same size to random neurons. If it moves the score too, the “fix” is noise.
Held-out, once. Score the locked edit on data it never saw. Count rows fixed and rows broken, and compute beforehand the smallest effect the test can detect.
Read the outputs. Benchmarks miss things. For chat, we read the answers side by side.
Test cases
Small task models of 13.5M to 88M weights, each with one known failure and a held-out set.
| Model | The failure | Kind |
|---|---|---|
| Date parser | Invents an hour for “haircut on friday” | one choice: “no time” vs an hour |
| Date parser | Reads “the day after tomorrow” as +1 day | one choice: a digit |
| Reminder tagger | Swallows “on friday” into the title | one choice: a tag |
| Text formatter | Adds too many paragraph breaks | a threshold, spread across the network |
| Chat model (MoE expert) | Answers loop until the length cap | one choice: when to stop |
| Invoice extractor (MoE expert) | Loops on the vendor field | copies the wrong span |
| Reminder writer | Writes stock words instead of the title | copying gap |
What happened
Held-out numbers, scored once.
| Model | Edit · weights changed | Before → after | Fixed / broke | Random edit | Worked? |
|---|---|---|---|---|---|
| Date parser: “no time” | copy 5 neurons ×6 · 18,952 of 14.5M (0.13%) | invented hours 152 → 42 of 300 | 90 / 0 | 149 of 300 | yes |
| Date parser: “+2” | copy 10 neurons ×6 · 37,913 (0.26%) | 27 → 81 of 200 | 54 / 2 | 26 of 200 | yes, with a side effect |
| Reminder tagger | copy 10 neurons ×4 · 23,809 of 14.95M (0.16%) | 251 → 280 of 326 | 34 / 5 | 250 of 326 | yes |
| Text formatter | lower 2 output biases · 2 numbers | paragraph F1 .671 → .757 | 1,156 / 309 words | ±0.01 F1 | threshold only |
| Chat model | shift the stop token’s row · 1,280 numbers | loops 150 → 63 of 265 | 88 / 1 | 155 of 265 | yes, shorter answers |
| Chat model, the paper’s method | remove the “repeat” direction · 54k–970k entries | dropped on practice data | – | – | no |
| Invoice extractor | zero 5 neurons · 6,078 (0.014%) | 755 → 753 of 1,600 | 24 / 26 | 753 of 1,600 | no |
| Reminder writer | copy 10 neurons ×4 · 22,717 (0.15%) | 169 → 167 of 326 | 1 / 3 | 169 of 326 | no |
Formatter: n=71,183 words over 509 documents; list F1 also rose, .564 → .620. Reminder writer: this test could only detect a change of 36 rows or more. Every edit on a ternary model kept its tensors ternary.
Two more things showed up. The “+2” fix also turned 10 of 142 synthetic “N minutes ago” rows into “−2”, a case the practice set barely covered. And copying neurons was cleaner than any threshold change: it stayed inside the model’s own values and broke few rows.
The loop story
Loops were the paper’s headline, so we expected the chat model to be the easy case. It wasn’t, and why it wasn’t is the most useful result here.
The paper’s edit failed. We found the “repeat” direction in the chat model’s MLP outputs and removed it from the output columns, as the paper did. On practice data it lowered good tokens as much as loop tokens, so it damaged normal text without breaking the loops. The reason: inside a loop the model is near-certain. The repeated token has median probability 0.996, and stopping has 0.0000 (n=7,380 positions). There is no decision left to tip.
The real error comes earlier. At the token just before an answer starts repeating, the chance of stopping is 0.0000013 (n=155). At real endings it is 0.85 (n=95). The model runs past its own end and then falls into repetition. That makes the loop a single missed choice, the same kind as the date and tag failures.
The cheap test agreed. Pushing the stop token up a little at generation time cut practice loops from 155 to 73 of 276, and the instruction-following score barely moved (37 → 36 strict passes).
What worked as a weight edit: shifting the stop token’s row in the embedding along the model’s average hidden direction. It changes 1,280 numbers and acts like a fixed push toward stopping. The paper’s rules would have banned this edit. The version that stays inside their rules, copying the neurons that signal “stop” inside the expert, helped only partly (150 → 120 of 265).
| IFEval held-out, 265 prompts | Loops | Hit length cap | Strict pass | Ends mid-sentence |
|---|---|---|---|---|
| original | 150 | 130 | 43 | 7 |
| stop shift, moderate | 63 | 19 | 36 | 36 |
| stop shift, strong | 26 | 5 | 38 | 89 |
| random shift, same size | 155 | 133 | 43 | 6 |
| copy “stop” neurons ×3 | 120 | 98 | 44 | 13 |
The model is a ternary MoE chat expert (88M) on our 8K foundation, greedy decoding, up to 1,024 new tokens. Smallest detectable change: 32 rows for loops, 22 for strict passes. The pre-registered rule picked the strong dose; the moderate dose was declared as a second arm before any test scoring.
On 100 real user prompts (WildChat), loops fell from 62 to 31 with the moderate shift. A random shift of the same size gave 64.
Read as a user would. The edit keeps each answer’s opening and stops it at a paragraph end, before the repeat. It does not make the content better. The strong dose cuts answers mid-thought, and the instruction score did not notice; counting mid-sentence endings and reading the answers did. That is why the moderate dose, not the rule’s pick, is the usable one.
“What is an inverter”, original (251 tokens):
An inverter is a type of inverter that is used to convert a circuit from one mode to another. It is a type of inverter that is used to convert a circuit from one mode to another. Inverters are used to convert a circuit from one mode to another, and they are often used in combination with other inverters to achieve the desired output. Inverters are typically used in a variety of applications, including: 1. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems. 2. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems. 3. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems. 4. Power conversion: Inverters are used to convert a circuit from one mode to another, and they are often used in power conversion systems. Overall, inverters are a powerful tool for converting a circuit from one mode to another, and they are widely used in various applications.
Moderate stop shift (79 tokens):
An inverter is a type of inverter that is used to convert a circuit from one mode to another. It is a type of inverter that is used to convert a circuit from one mode to another. Inverters are used to convert a circuit from one mode to another, and they are often used in combination with other inverters to achieve the desired output.
A math prompt from IFEval: “11 results have an average of 51. The first five have an average of 49, and the last nine have an average of 52. What was the sixth result?” The original loops to the 1,024-token cap:
The sixth result is: 11 results have an average of 51. The last nine have an average of 52. So, the sixth result is: 11 results have an average of 51. Now, we need to add the results of the last nine to the total number of results: 11 results have an average of 51. The sixth result is: 11 results have an average of 52. Adding the results of the last nine to the total number of results: 11 results have an average of 52. Therefore, the sixth result is: 11 results have an average of 52. So, the sixth result is: 11 results have an average of 52. The sixth result is: 11 results have an average of 52. [… these last two lines repeat until the cap]
Moderate stop shift (6 tokens): The sixth result is: — strong (4 tokens): The sixth result is
This is the paper’s loop failure on a math problem. The paper’s loops hid answers the model could reach. Here nothing was hidden: with the loop gone, the answer stops after “The sixth result is”. The correct answer is 152.
The rule, and the test to run first
Editable when the failure is one choice at one token, and the right token is one the model already produces elsewhere.
Not editable when the model writes or copies the wrong content over many tokens. Small edits can’t add a missing skill.
The test (minutes, no editing): on practice data, push the right token’s score up a little at generation time. If that alone fixes the failure, a weight edit can do it too. If there is no single token to push, don’t try.
| One choice at one token | Content over many tokens | |
|---|---|---|
| Cases | date “no time”, date “+2”, tagger, formatter, chat stop | invoice span, reminder title |
| Push-the-token test | fixes it (chat: loops 155 → 73 of 276) | no single token to push (invoice: the “close” choice explains 19 of 1,000 loops) |
| Edit result | 5 of 5 fixed | 0 of 2 |
Which edit to use, once it passes: if a few neurons carry the decision and fire too weakly on the failing rows, copy them into little-used neurons of the same layer; that keeps a ternary tensor ternary. If failing and passing cases look the same inside the network (formatter: the largest neuron gap was 0.094), shift a threshold instead.
A side question: can an edit make a model ternary?
The same day we asked whether direct edits, again with no gradients and no training, could turn a dense model into a ternary one. On SmolLM2-360M-Instruct, the best pure-ternary edit closes 80.0% of the gap between the dense model and plain rounding, measured in log-perplexity on WikiText-2 (n=148 chunks): perplexity 128, against 12.7 dense and 1.3M for plain rounding. Keeping 2 of 32 layers dense closes 85.6% (perplexity 67). None of the 21 ternary versions answered any of 5 chat prompts correctly (0 of 5 each). So: yes, the weights can be made ternary by editing alone, but not into a usable chat model.
Limits
One editor and one run per model. The editor wrote some of the practice and test probes. The instruction-following set, the real user prompts and one handwritten reminder test set were written by others, which makes those the strongest held-out evidence.
The instruction score fell from 43 to 36 with the moderate stop shift (4 won, 11 lost). This test could only detect a change of 22 or more, so a real loss of that size can’t be ruled out.
The stop shift changes a shared tensor (the tied embedding), so in a model with several experts it affects all of them. It was not tested on other routes. The ternary-only neuron copy stays inside the expert.
None of these edits has been exported to Core ML or driven in an app.
Links
MIT CSAIL, “Sparks of Weight-Space Programming” — rikiyat.github.io/blog/weight-space-programming
Announcement on X — x.com/rikiyatakehi
This write-up on X — x.com/TslShahir
Get new experiments
Occasional notes when we publish. No spam.