Recent Developments in LLM Architectures: KV Sharing, mHC & Compressed Attention
— by Vivax
A wave of new open-weight LLMs — from Gemma 4 to DeepSeek V4 — share one obsession: making long-context inference cheaper without simply shrinking the model.
As reasoning models and agentic workflows keep more tokens in play for longer, the cost of attention — KV-cache size, memory traffic, and compute — has become the dominant constraint on large language models. In a widely-read analysis of the open-weight releases from April and May 2026, Sebastian Raschka, author of 'Build a Large Language Model From Scratch,' argues that the most interesting trend this year is not bigger models but architectural tricks that make long-context inference cheaper. Most of these changes look like small tweaks in an architecture diagram, yet several are intricate redesigns of how the transformer block handles memory and attention.
Gemma 4 leads with two ideas. Cross-layer KV sharing lets multiple attention layers reuse the same key-value cache instead of each storing its own, cutting the memory that dominates long-context inference. To make up for the capacity that sharing gives away, Gemma 4 adds per-layer embeddings — extra, layer-specific embedding capacity that can be streamed from CPU or storage — so the model stays expressive without inflating the KV cache. The net effect is a smaller memory footprint for the same context length.
Other releases attack the same problem from different angles. ZAYA1-8B moves attention into a compressed latent space with a compressed convolutional attention (CCA) design, shrinking the cost of each attention step. Laguna XS.2 treats attention as a budget, tuning how much attention capacity each layer gets rather than treating every layer the same. And DeepSeek V4 combines two heavier changes: a constrained form of residual-stream mixing the author calls mHC (mixed-head compression), plus a compressed long-context attention scheme — layering efficiency tricks on top of the multi-head latent attention DeepSeek introduced earlier.
The throughline is clear. The basic recipe is still the original GPT-style decoder-only transformer, but the attention machinery is being upgraded, specialized, and compressed for longer contexts and cheaper inference, while qualitative model quality seems driven mostly by data and training recipes. The trade-off is complexity: a basic transformer block was once 50–100 lines of PyTorch; these attention variants can 10× that, making it harder to reason about how the pieces interact — even though they reduce, not increase, runtime cost. Notably, transformers remain the status quo; none of these releases replace them.
We follow this work closely at Vivax because long context is not an abstract benchmark for us — our native medical LLM has to reason over long, messy patient histories spanning many visits, documents, and signals. The cheaper and more memory-efficient long-context attention becomes, the more feasible it is to run grounded, decision-grade clinical AI affordably and, where it matters most, on-premise inside a hospital's own walls. The architecture race for efficient long context is, indirectly, a race toward practical medical AI.