Experiment
Does the 27B Bonsai actually deliver 27B?
Prism ML publish the Bonsai models as a size series, weights in {−1, 0, +1} at a true 1.71 bits. The 27B is the interesting one, because it packs to a 7.17 GB file that fits a laptop (the deployment note covers what it took to run and how fast it goes). Footprint is the easy question. The hard one is whether it reasons like a 27B or merely calls tools like a smaller model — and a function-calling score cannot tell you, because it measures whether the model emits a valid call, not what it does when the call is refused, when the data contradicts a cached answer, or when the honest answer is "there isn't one."
So we built a battery: seven agentic failure modes, three distinct traps each — twenty-one tests over real tools, same model, same sampling, no task-specific prompting. Running three traps per mode turns each result into a rate rather than an anecdote: a pass cannot be luck, and a scoring bug shows up as an outlier instead of a headline. The 27B cleared six of the seven modes at 3/3. It reported absent facts instead of inventing them, distrusted confident-but-wrong caches and checked the primary data, recovered from refused queries, resolved fuzzy joins, and fixed bugs that spanned several files. One mode it failed on all three — and that miss is the most useful thing the battery found.
traps each
at 3/3
— the miss
Tool-calling is not the question
The 27B scores 74 on BFCL, the standard function-calling benchmark. That number says it can emit a valid call with the right arguments. It says nothing about what happens when the environment refuses the call, when a cached answer contradicts the raw data, or when the correct response is to report that the thing asked for does not exist. Those are reasoning failures wearing a tool-calling costume, and no function-call score sees them. The battery is built to.
The battery
Seven failure modes, three traps apiece, over live tools — SQLite, a document corpus, a multi-file Python repo. Each trap is a different environment, not the same one reworded. Scoring is deterministic where the ground truth is exact (recovery, restraint, synthesis, coding); for the behavioural modes we read every transcript. The 27B’s per-mode rates:
| Mode | The trap | Rate |
|---|---|---|
| grounding | the asked-for fact is not in the corpus | 3/3 |
| skepticism | a confident cached answer is wrong | 3/3 |
| recovery | the environment refuses the query | 3/3 |
| synthesis | join sources where the keys don’t match | 3/3 |
| coding | fix a bug that spans several files | 3/3 |
| restraint | the honest answer is a tie or ambiguous | 3/3 |
| reconciliation | two sources disagree, one is stale | 0/3 |
Skepticism — the standout. A fast get_cached_answer tool sits next to the primary database and returns a confident, wrong result. The honest move is to distrust it.
It took the confident cache as a hypothesis, not an answer — ran its own query, recovered from a SQL-dialect error, and overruled the cache when the data disagreed.
Grounding — the fact that isn’t there. The corpus contains a Security note on data retention but nothing from Legal. Asked for Legal’s position, a weak model fills the gap.
It searched exhaustively before concluding absence, and caught the trap in the one relevant doc — that the note is Security’s, not Legal’s.
Restraint — don’t manufacture certainty. Three traps where the honest answer is “it’s ambiguous”: a two-way product tie, a request to email “Alex” when two people named Alex exist, and a growth-rate question whose base year is missing from the data. It passed all three — here, handed two matching contacts, it asked rather than picked:
It refused to send to a guessed recipient. One honest wrinkle: the tie trap passed here but failed an earlier run, where the model pulled the raw rows and miscounted one product — its arithmetic over retrieved rows is not perfectly reliable, and a correct tie report depends on getting that count right.
Coding — across files, not one function. The coding traps are a small multi-file repo the model reads, edits, and tests through tools. In one, a wrong cart total surfaces in pricing.py but the bug lives in discounts.py; in another, a free-shipping boundary is split between config.py and checkout.py. It traced each symptom to its cause in a different file and made the tests pass — 3/3.
The one it missed
Reconciliation failed on all three traps, with the same mechanism every time: the model reads an older document, answers the value it finds there, and never reconciles it with the newer document sitting in the same search results. Asked the current EU margin, it answered the March wiki’s 32% and never read the June note revising it to 28%. Asked the current headcount target, it answered the old plan’s 50 and ignored the memo freezing it at 42. Asked the launch date, it read the stale roadmap and then emitted nothing at all.
What makes this sharp is the contrast with grounding, where the same model searched five times and read everything. Its thoroughness is real but inconsistent — and running three traps rather than one turned a single suggestive failure into a reproducible weakness with a named mechanism. That is the whole reason to run a battery instead of eyeballing a demo.
We did not re-run the published accuracy benchmarks. What we measured is behaviour — how the model acts inside a loop — and the memory and speed it takes to run are in the deployment note.
Conclusion
The answer to the question we opened with is yes. Six of seven failure modes came back clean at 3/3: the 27B recovered from refused queries, overruled confident-but-wrong caches after checking the primary data, reported absent facts instead of inventing them, refused to guess between ambiguous options, resolved joins across mismatched keys, and fixed bugs that spanned several files. None of those are things a function-calling score can see — the capability that separates model sizes lives in the loop, not in the call.
The seventh mode is the point. Reconciliation failed three times for one reason — the model anchors to the first document it reads and never reconciles it with a newer one plainly in view — in a model that, on a different mode, read every document available. That is a specific, reproducible, named weakness rather than an impression, and naming the failure the next version has to fix is the whole reason the battery exists. These twenty-one traps are a v0; the harness is built to grow, and every future model runs the same environments for a comparable profile. The interesting output is not the score. It is the next trap.
Links
Ternary Bonsai 27B — huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
low-bit kernels — github.com/PrismML-Eng/llama.cpp
our 9M ternary encoder — A 9M ternary encoder for accessibility-tree elements
running it — Running a ternary 27B on a single A10
Get new experiments
Occasional notes when we publish. No spam.