Zyphra Hierarchical Attention: The End of Quadratic Attention?

Zyphra Hierarchical Attention: The End of Quadratic Attention?

Every long-context model carries the same quiet tax. When it generates a token, attention compares that token against everything that came before, so doubling the context roughly quadruples the work. The workarounds each give something up: fixed-state methods like sliding windows and linear attention compress the past into a bounded summary that eventually overwrites information you still need, and sparse attention keeps the full cache but pays a indexer to decide what matters. The uncomfortable detail, published in cited work, is that near one million tokens that indexer consumes 60 to 90 percent of layer time. The attention got sparse, but the searching did not. ZHA is the name of the method that changes this.

Zyphra Hierarchical Attention, or ZHA, released with a technical report at the end of October, is a direct answer to that problem. ZHA builds a tree of recursively compressed key summaries over the KV cache and lets each query walk down the tree, expanding only the branches that look relevant. Because the search itself is hierarchical, the cost of finding the right tokens grows sublinearly with context length, which changes the asymptotics rather than the constants.

Two claims make ZHA worth attention. First, ZHA was distilled into frozen pretrained models with about 4 billion tokens of training and 81,920 new parameters, roughly 0.00024 percent of the base model. Second, on a 35B-parameter model it stays close to dense attention on reasoning, coding and retrieval benchmarks, and on some retrieval tests it beats dense while reading about 1 percent of the cache.

This article explains how the ZHA tree works, what the benchmarks actually show, where the speedups are real and where they are projections, and what the method changes about the long-context roadmap.

Why sparse attention had a hidden quadratic

The two existing families fail in complementary ways, and the failure modes are worth stating precisely.

Fixed-state approaches, including state-space models and sliding windows, maintain a bounded representation of history. The analogy Zyphra uses is taking notes on an ever-growing pile of books with a fixed number of pages: eventually new information overwrites old information regardless of its future value. That is a fundamental ceiling, not an engineering nuisance.

Sparse attention keeps everything and selects. But the selection mechanism is the catch. Flat indexers score every past token against the query, so the indexer cost grows with full context length even when the final attention touches only a sliver of it. Chunked indexers reduce the constant but still scan every chunk at one resolution, which is linear, not sublinear. In both cases the quadratic problem has been relocated, not removed.

Zyphra's starting observation is that dense attention models are already secretly sparse, and ZHA exploits exactly that sparsity. Measuring the number of tokens that account for a fixed fraction of attention probability, the size of that set grows sublinearly with sequence length, often following a power law on long-form fiction. If the actively retrieved set grows more slowly than the context, then a search structure that adapts to it can be sublinear in principle. ZHA is the construction of that idea.

How the tree is built and searched

The ZHA mechanism has two halves: a compressed index and a routing rule.

Building the index is recursive. The keys in the KV cache are split into contiguous chunks, and each chunk is compressed into a summary key. The summaries are then grouped and compressed again, level by level. With a chunk size of four, six levels cover 4,096 tokens per top-level node. Crucially, the full KV cache is retained at the leaves; the tree is a search structure laid over the memory, not a replacement for it.

The summaries are not averages. Averaging keys in a chunk destroys exactly the information a future query might need, because different queries care about different aspects of the same chunk. Instead, ZHA learns a small set of reference queries, and each one provides a tangent-plane estimate of the chunk's log attention mass. Taking the maximum over these planes approximates how much attention any future query would pay to the chunk. The summary is trained to answer one question: how relevant is this region likely to be?

Routing in ZHA is top-p rather than top-k. A query scores the top-level nodes, keeps the smallest set holding a fraction p of the estimated attention mass, expands only those children, and repeats until it reaches raw KV entries. The first token is always kept as an attention sink. The adaptive part matters: a single-fact lookup touches few branches, while a query synthesizing scattered facts expands many more. A fixed top-k budget forces a permanent compromise between those cases; top-p lets the compute follow the query.

That is the whole trick behind ZHA, and its elegance is that the same structure serves both goals: the tree bounds the search cost, and the retained KV cache preserves expressiveness.

What the benchmarks show

Zyphra converted Qwen3.6-35B-A3B with two ZHA tree geometries, a deep six-level tree and a wide single-level tree, both using 16 reference queries and p set to 0.95, and trained each with about 4 billion tokens on a frozen backbone. The comparison table also includes a MiniMax Sparse Attention indexer trained on 40 billion tokens, ten times the budget.

The headline rows, all self-reported:

• RULER-13 at 256K context: the wide tree scores 96.2 percent against 96.0 for dense attention, while the deep tree reaches 93.3. MiniMax's indexer drops to 77.1.
• LongBench v2: the deep tree hits 63.4 percent against 58.0 for dense, while reading about 1.1 percent of keys. The same test puts MiniMax at 49.3.
• SWE-bench Verified: 67.4 for the deep tree, 68.4 for the wide tree, 67.4 dense. Agentic coding survives the conversion intact.
• AIME 2026: 91.2 and 90.8 against 93.1 dense, a small reasoning cost.
• MMLU-Pro and GPQA Diamond: parity or slightly above dense.

Reading the ZHA table honestly, three patterns stand out. Retrieval is where sparse methods usually bleed, and ZHA bleeds least, including beating dense on one long-context benchmark. Reasoning costs a point or two on the hardest math. And the MSA comparison, where ZHA used a tenth of the training tokens and outperformed by 14.1 points on LongBench v2 at a similar key share, is the strongest evidence that the tree structure itself, not extra training, carries the result.

The geometry study strengthens the ZHA claim further. Across 28 tree configurations, RULER accuracy stayed stable when attending to 10 to 30 percent of keys, and at fixed compression, deeper trees performed better than shallower ones. Depth, the thing that would naturally be feared as a bottleneck, is not one. A four-parameter model predicts RULER accuracy from geometry with a 1.44-point error, which suggests the design space is navigable without exhaustive search.

A query's path through the tree, step by step

Abstractions get clearer with a concrete trace, so consider a coding model with ZHA installed, holding a 256K-token repository in its KV cache, asked to fix a bug that references a utility function defined ten thousand lines earlier.

The query vector first meets the top level of the tree, where each node summarizes 4,096 tokens. Sixteen reference queries score roughly 64 top nodes, comparing the query against tangent-plane estimates of each region's attention mass. Top-p routing keeps the smallest set covering 95 percent of that estimated mass. For a bug fix citing a specific function, most of the probability mass concentrates on a handful of regions: the file containing the function, the call sites, and the sink token.

The selected nodes expand. Each top node holds summaries of 1,024-token regions, then 256, then 64, then 16, then 4, until the search reaches raw keys. At each level the same rule applies, so irrelevant regions are discarded in bulk at the top, where discarding is cheapest, and only the promising slivers of context are examined in detail. The final attention runs over the surviving raw keys, exactly as dense attention would, just over a few hundred tokens instead of a quarter million.

The result the benchmarks measure is that this trace touches roughly 1 percent of the cache while producing answers indistinguishable from reading everything, on tasks where a single missed function definition means a failed test. The deeper point is architectural: the tree is built on the fly as the context grows, so the index is never a preprocessing step and never a separate system. It is the model's own memory, indexed by the model's own learned queries.

The distillation trick is the real product

For most teams, the conversion method matters more than the architecture, because it changes who can use it.

ZHA does not require pretraining a model from scratch, which is the property that makes it a product rather than a paper. The backbone stays frozen, a KL loss teaches the reference queries to reproduce the dense model's attention mass per chunk at every level, and 3.98 billion tokens over 1,900 steps at 32K context was enough, including for evaluations at 256K. The pretrained model effectively teaches the indexer how to search its own memory.

The economics follow directly. Any capable open model can gain sublinear long-context retrieval for the cost of a brief distillation run, without touching the weights that carry its reasoning and language ability. The reported scaling behavior also matters: models under 1 billion parameters struggled during conversion, with results improving sharply at 4 billion and above, so the technique currently belongs to the mid-size class and up.

The operational simplicity is worth dwelling on, because long-context techniques have historically demanded enormous sacrifices. Adopting a new architecture has meant pretraining from scratch, an expense only a handful of organizations can justify. ZHA's retrofit path inverts that: the expensive part of the model, everything that makes it smart, is untouched, and a distillation run measured in billions of tokens, rather than trillions, adds the retrieval structure. The 81,920 parameters are the tell. A technique that modifies a 35B model with fewer parameters than a small embedding table is not changing the model; it is teaching the model to navigate what it already knows.

There is a precedent worth noting in how ZHA compares with other groups' paths. DeepSeek's sparse attention required continued pretraining of an existing model, and MiniMax's indexer used a 400 billion token continued-pretraining recipe. A four-billion-token retrofit on a frozen backbone is a different accessibility class, and if the results replicate, it is the difference between a research result and a technique any lab with a few GPUs can apply.

The speed claims, separated carefully

This is where honest reading of ZHA matters most, because its most impressive numbers are projections.

No optimized kernels have been released. The speed figures come from a modeled H100 roofline for one layer at batch size one. In the lightweight configuration, the model has ZHA beating dense attention near 48K tokens and beating MSA near 0.48 million tokens. At one billion tokens, projected speedups reach 373 to 417 times dense attention and 52 to 58 times MSA. At 256K, which is where current models actually operate, the modeled gains are modest: 1.93 to 2.28 times dense at 92.7 to 96.4 percent RULER accuracy.

So the fair summary of ZHA's speed story is layered. At today's context lengths, ZHA is a modest speedup with essentially no quality loss. At million-token contexts and beyond, the asymptotic advantage becomes dramatic, but that regime is modeled, not measured, and routing costs that are negligible at 256K have more room to bite as depth grows. The paper's claim of subquadratic, not sublinear, scaling is the correct frame: the method changes the curve, it does not repeal it.

The missing kernels deserve emphasis, because they cut both ways. Without released implementations, no external team can verify the modeled latencies, and the history of efficient attention is full of methods whose paper speedups shrank under real kernel engineering, where memory movement dominates arithmetic. The roofline model assumes the routing overhead stays small relative to the attention it saves, which the 256K figures support but the billion-token figures have to assert. The practical instruction for practitioners is to treat the 2x at 256K as the only actionable number today and to watch for kernel releases as the signal that the larger claims have become testable.

The Nemotron experiment adds a boundary condition. On a NoPE-based model, deep-tree RULER-13 fell 25.3 points below dense, a reminder that the method's retrieval guarantees lean on positional structure that not every architecture provides.

What to verify before adopting it

The honest evaluation path for ZHA is specific, because the claims split cleanly into measured and modeled.

Replicate the distillation on your own model class: the recipe is 4 billion tokens at 32K context with a frozen backbone, and the reported sensitivity to model size below 1 billion parameters means small models need their own evidence. Run your own retrieval suite, not just RULER, because RULER-style needle tests are exactly where sparse methods have historically looked better than they behave on tangled, multi-document questions. Test agentic workloads end to end, since SWE-bench parity is the single most decision-relevant number for coding agent teams and it deserves confirmation outside Zyphra's harness. And wait for kernels, because every speed figure beyond 2x at 256K is currently a projection from a roofline model, and a projection is a hypothesis with equations.

None of these caveats weigh against the core result. They define the distance between a strong paper and a deployable technique, which is a distance every efficient attention method has had to cross.

What ZHA changes about the long-context roadmap

If the ZHA results hold under replication, the practical implications are concrete.

For teams running long-context workloads, from repository-scale coding agents to document intelligence, the near-term value is a retrofit path: take a strong open model, spend a small distillation budget, and get 256K-class retrieval at a fraction of the attention cost. For model developers, the geometry study suggests depth is a tunable resource rather than a risk, which makes the sublinear scaling claim usable rather than theoretical. And for the field, the result sharpens the design question that fixed-state methods have been dodging: the goal is not the smallest state, but a state that grows just slower than the context.

The connections to the rest of the agent stack are direct. Long-context models and long-lived agents solve the same problem from different ends, and the infrastructure argument in persistent sandbox compute for agents is the systems-side twin of ZHA: keep everything, pay only for what you touch. The evaluation lesson also echoes what we found in the inference caching guide, where the biggest wins came from not recomputing what was already computed. And for coding agents specifically, a 256K window that actually works at 96 percent retrieval changes what agent workspaces can hold in memory rather than in tools.

Whether this ends quadratic attention is the wrong question. What ZHA demonstrates is that the search over context can be made as adaptive as the attention over it, and that the demonstration can be retrofitted onto models teams already run. The end, when it comes, will look like this: not one paper that kills the quadratic, but a sequence of results, of which this is a serious one. The question to watch is no longer whether subquadratic attention can preserve quality on retrieval benchmarks. That question ZHA has largely answered. The question is when the kernels, the hardware patterns, and the independent replications arrive to make the million-token regime a deployment target rather than a projection, because every part of the agent stack, from coding assistants to document intelligence, is waiting on exactly that.

Sources

• Zyphra Hierarchical Attention, Zyphra research blog: https://zyphra.com/our-work/zyphra-hierarchical-attention
• ZHA technical report: https://arxiv.org/abs/2610.42112
• Qwen3.6 model family, Qwen on Hugging Face: https://huggingface.co/Qwen