A 9M ternary encoder for accessibility-tree elements

The problem

An agent acting on a web page first has to resolve a natural-language instruction to a concrete element, and the same instruction maps to different element types across sites — “check out” might be a button here, a link there, a menu item elsewhere. The raw tree is enormous — most of it irrelevant to any one instruction:

One live amazon.in shoes-search page: 13,444 DOM nodes, ~1,800 interactive elements. For a given instruction the filter surfaces a handful — “search box” resolved to a single element at 0.97 confidence. Bars are log-scaled.

Feeding the whole DOM to a large model in-context is the obvious approach and a wasteful one — thousands of nodes of context for what is a narrow choice. So the filtering runs first, narrowing the page to the few elements an instruction could plausibly mean. That filter is multilayered; this model is the layer that classifies each element — small and fast enough to run on every element of every page (~2 ms on the Neural Engine, a bundle under 10 MB).

Method

Five steps. The choice that matters is ordering: the label set is discovered from the data before the model is trained, not assumed up front.

1. Corpus. Chrome DevTools Protocol accessibility-tree snapshots from a ~7,000-page dataset (corpora walk-v3, cdp-survey, and webui-7k). Each node is rendered to a fixed-grammar feature string — role, accessible name, URL-path prefix, and parent role — and tokenized with a 4,096-token byte-level BPE to a length of 128. The URL-path prefix is treated as a first-class feature: /cart, /checkout, /login, /p/ are diagnostic for several link and action classes.
2. Scope. The role-based interactability filter rejects 72.7% of nodes (mostly text fragments). To decide whether those rejected nodes need a learned classifier of their own, we measured the rate at which they carry a user-action event listener (click, mousedown, keydown, submit), via DOMDebugger.getEventListeners on a sample: 2.54%. On that basis the model is scoped to the role-recognized elements (INTERACTIVE_ROLES) only, and the residual cases are covered by three deterministic rules — label[for] click-forwarding, native <details>/<summary>, and trusting a DOM-declared role attribute over a generic accessibility role.
3. Label discovery. Over ~342K candidate feature strings we run TF-IDF + k-means with k=50, then ask an LLM (GPT-4.1-mini) to label the 50 centroids rather than the individual points — two orders of magnitude cheaper, and consistent because one model assigns all labels in one pass. The clusters are then merged, dropped, and relabeled by hand into 25 classes. The data does not group the way a hand-written taxonomy would: a single “button” splits into eight intent-aligned actions (cart_action, buy_action, submit_action, subscribe_action, book_action, save_action, dismiss_action, auth_action), and “link” splits into five by destination semantics, where the URL-path prefix is the dominant signal.
4. Classification by centroid, not softmax. The encoder outputs a 128-dim L2-normalized embedding. After training, a centroid is computed per class as the mean of its training embeddings and frozen as an 11 KB array; classification is the nearest centroid by cosine. There is no learned label head, so the taxonomy can be re-bucketed (the 21-class v7 became the 25-class v8) by recomputing centroids alone, without retraining the encoder.
5. Ternary QAT from the start. Every linear layer is ternarized in the forward pass — weights in {−α, 0, +α}, threshold 0.7·mean(|W|), per-layer scale α, straight-through estimator — with an FP32 shadow kept for re-export. Quantization-aware training runs from epoch 0, no FP32 pretrain. The shipped weights are 2-bit palettized for the Neural Engine.

Model and training

An 8-layer bidirectional transformer encoder, d_model 256, 4 heads, RoPE positions, RMSNorm and a SwiGLU FFN (hidden 1,024), mean-pooled over non-pad tokens and projected to 128 dimensions on the unit sphere — about 9M parameters (36 MB in FP32). The objective is supervised contrastive loss (τ = 0.07), averaged only over anchors that have an in-batch positive, which matters under heavy class imbalance so tail-class anchors still receive gradient. AdamW, peak LR 3e-4 cosine-decayed, batch 256, 20 epochs. Depth helped more than width: 4 to 8 layers improved accuracy, while widening to 320 dimensions gave no measurable gain.

The ternary constraint did not bind. Validation accuracy before and after ternarization was identical within noise — 96.3% / 96.3% on the webui-7k validation split — so the Neural-Engine deployment comes at no accuracy cost.

Results

On a true site-holdout — centroids computed from training-site rows only, then 2,547 elements from 14 unseen sites classified by nearest centroid — the shipped v8tv2 checkpoint reaches 84.73% microaccuracy. The result is sharply uneven across the taxonomy:

By site the spread is just as wide — from 60% on content-dense pages like GitHub to 98–99% on Substack, Supabase, and docs. The weakest class is the one that matters most for commerce routing (cart_action), which is why the training set includes mined hard negatives for it.

The confidence signal is the cosine to the predicted centroid — no calibration head needed: high is reliable, middling is a candidate for a cloud-model re-rank, low means abstain.

Deployment

The shipped artifact is a four-file bundle treated as a unit: the 2-bit CoreML weights (2.53 MB), the BPE vocabulary (~260 KB), the 25×128 centroid array (~11 KB), and the label names. End-to-end per element is about 1.2 ms — ~0.10 ms to tokenize, ~0.95 ms for the Neural-Engine forward (CoreML compute units set to CPU-and-Neural-Engine), ~0.15 ms for the centroid cosine and argmax — so a 300-element page is classified in roughly 360 ms, entirely on-device.

How Haku uses it

The model is not the sole decider — it re-ranks on top of Haku’s element search, which scores each candidate by how well its text matches the instruction (an EmbeddingGemma cosine similarity). That text score is blind to intent, so two look-alike buttons tie; the classifier breaks the tie. When the agent scans a page, every actionable element is classified into one of the 25 classes and cached; the instruction is run through the same centroids for its own top classes. The classifier then contributes in two ways:

• Class-agreement re-ranking. A gate fires only when the instruction’s class signal is sharp — top-1 centroid confidence ≥ 0.85 and a margin of ≥ 0.15 over the third. When it fires, elements whose predicted class is among the instruction’s top two are boosted (×1.25). This is one factor in a product of signals (text relevance, container, capability, URL), not an override, so a confident textual match still wins.
• Label enrichment. Above a 0.65 confidence floor, a predicted class is folded into the element’s label — cart_action becomes “add to cart”, auth_action becomes “sign in” — which feeds the text matcher. Structural classes (headings, landmarks) are deliberately left un-enriched so they don’t pollute action queries.

Two properties fall out of this. It degrades gracefully — when it is unsure, the gate doesn’t fire and ranking falls back to text relevance. And it is deterministic — a feature string maps to a fixed embedding and a fixed nearest centroid, so the same page routes the same way every time.