Understand the two technical improvements made by Kimi k3. Kimi Delta Attention a hybrid linear-attention mechanism the company says delivers over 6x faster decoding at long context, and Attention Residuals, a drop-in replacement for standard residual connections that improves training efficiency.

Kimi Delta Attention (KDA) refines the gated delta rule with fine-grained, channel-wise gating and runs in a 3:1 hybrid with full attention. It cuts KV-cache memory by up to ~75% and hits up to ~6× faster decoding at 1M-token context.

Attention Residuals work along depth (how information moves between layers).

Both are drop-in replacements for standard transformer components, but they act on two different axes of the model. KDA works along sequence length (how information moves between tokens).

Attention Residuals work along depth (how information moves between layers). Here’s the first.

The core problem KDA solves is that standard softmax attention makes every new token look back at every earlier token, so the memory it must keep (the KV cache) and the work it does both grow with context length. KDA replaces that with a single fixed-size memory that each token updates and reads.

Two details make KDA more than just “linear attention.”

1. the delta rule: instead of blindly piling every new key–value into memory (which causes older associations to blur together), it writes a correction — it subtracts the stale value tied to a key and writes the fresh one, keeping the fixed memory precise.

2. the fine-grained gating, the forget gate α decays memory per-channel rather than with one global knob, which is the refinement over Gated DeltaNet and Mamba2. Because a fixed-size state can still miss exact long-range lookups, KDA isn’t used alone — it’s interleaved with full attention layers in a 3:1 ratio (three KDA layers per one full-attention layer), which is what recovers global retrieval while still cutting KV-cache memory by up to ~75% and delivering up to ~6× faster decoding at 1M tokens.

Now the second mechanism, working on the depth axis. In a standard transformer, each layer adds its output back to the input through a plain identity skip (x + f(x)). In very deep stacks — and K3 is very deep — that single residual stream can get saturated, so later layers struggle to reach information from much earlier ones and gradients weaken. Attention Residuals replace the plain add with a residual that can attend across layers.

KDA changes how information moves sideways across the sequence (fixed memory instead of a growing cache → the speed and long-context win), while Attention Residuals change how it moves upward through the layers (cross-depth residual instead of identity → the training-efficiency and depth-scaling win). Together they’re why a 2.8T-parameter model can serve 1M-token context at the price point that rattled the market.