Same model, new post-training. RL with rule-based rewards on V3-Base, no supervised reasoning data to start.
V3's own abstract concedes the point: MLA and DeepSeekMoE "were thoroughly validated in DeepSeek-V2." So when this talk explains MLA, it is reading V2. When it explains FP8 and the all-to-all kernels, it is reading V3.
Contents
Mixture of Experts
01Multi-head Latent Attention
Multi-Token Prediction
DualPipe
02FP8 training
Section 1
MLA
Attention patterns
Every head, in every layer, builds one square matrix of token-to-token affinities, then uses it to move information between tokens.
What one tile is
One head's affinity matrix. GPT-2 has 12 layers × 12 heads = 144 per forward pass.
Its side length
Exactly the sequence length, also called the context length. Grow the context, grow every tile.
What it does
Moves information between token positions and the model's residual stream.
DeepSeek-V3 / R1
61 layers × 128 heads = 7,808 tiles. Same concept, two orders of magnitude more of it.
Visualization: Welch Labs. The triangular shape is causal masking. No token may attend to one that comes after it.
Queries and keys
A layer receives a T × C matrix, one row per token. Two learned matrices turn it into queries and keys.
Head dimension is written as 64 here to match small-model convention; DeepSeek-V3 uses 128 per head, across 128 heads.
Matching keys to queries
A query row advertises what a token wants. A key row advertises what a token offers. When they point the same way, their dot product is large.
Nothing here is hand-specified. W_Q and W_K are learned, and "modifier" is just a pattern a head settled into during training.
Q·Kᵀ, mask, softmax
One matrix product gives every pair. Then mask the future, scale, and softmax each row.
The side length of this matrix is the context length, which is where the whole O(n²) budget goes.
The value matrix
The pattern says how much each token should listen to each other token. A third projection says what actually gets passed along.
Each output row is a blend of the value rows that token's pattern pointed it toward.
Multiple heads
A layer runs many in parallel, each free to specialise. One tracks adjectives, another looks for earlier instances of the same token.
This is the core of every modern language model, and where a large share of the compute goes.
MHA, end to end
Stacked outlines mean one per head.
10¹² dot products
Context windows reach a million tokens. Attention scales with the square of that.
No model does this. The next two slides are the trick that avoids it.
One decode step, T → T+1
Red marks everything the new token touches.
The mask discards the upper triangle anyway, so the entire cost of emitting one token is a single row of scores against the whole history.
Compute with and without a KV cache
Without a cache, every new token rebuilds K and V for the entire history, quadratic in tokens generated. With one, each step does a constant amount of new work.
The work does not disappear. It moves. What was arithmetic is now memory traffic, and the next slide is the size of that bill.
What the KV cache costs
The cache holds K and V for the full history, for every head, in every layer.
Three answers follow: share the K and V across every head, share them across groups of heads, or compress them.
MQA: one K and V for the whole layer
Same machine as before. The only change is upstream. W_K and W_V produce a single pair, reused by every query head.
The cache shrinks by exactly the head count. DeepSeek-V2's ablation found MQA trailing MHA badly: 37.9 vs 45.2 on MMLU at 7B.
GQA: one K and V per group
The compromise Llama 3 ships. Split the heads into n_g groups; each group shares one K/V pair.
Both MQA and GQA buy memory the same way: by deleting capacity. They sit at two points on one line, and MLA is the attempt to leave that line.
MLA: cache one latent, rebuild K and V
W_DKV is per layer and shared. W_UK,ᵢ and W_UV,ᵢ are per head. Only the red tensor is ever stored.
MHA's per-head W_K : C × d has become a factorization: C × r shared, then r × d per head. The cache sits in the shared middle, so it no longer depends on H.
The absorption
Both up-projections are frozen after training and neither depends on the token, so each can merge with a frozen neighbour. Nothing is approximated.
Same trick twice: a fixed matrix caught between two other fixed matrices is not a computation, it is a change of basis you can precompute.
MLA, absorbed
Queries are projected into latent space and score straight against L_KV. The pattern then weights the latent itself, and one merged matrix converts back. Red is the cached tensor, as before.
Cost of the swap: ~4× the MACs, ~64× less HBM traffic, arithmetic intensity up ~256×. And all H heads now share one L_KV, so their scores batch into a single GEMM instead of H separate GEMVs.
Cache size: MHA, GQA, MQA, MLA
Values cached for one token across all 61 layers, at DeepSeek-V3's shape: 128 heads, 128 dimensions per head.
Quality labels are DeepSeek-V2's own (Table 1). MQA and GQA sit on a line trading quality for memory; MLA leaves that line.
FlashMLA
DeepSeek released one publicly two months after V3.
github.com/deepseek-ai/FlashMLA · February 2025
Section 2
FP8 mixed precision training
Hopper: the GPU and one SM
Each processing block has its own scheduler and its own slice of the register file. Shared memory is the one resource all four blocks use.
Software structures
None of these are physical. You choose all of them except the 32.
Block sizes are therefore multiples of 128. Hopper GEMM kernels usually pick 384: one producer warpgroup issuing loads, two consumer warpgroups issuing WGMMA.
Scheduling, per processing block
A warp is assigned to one block for life. What varies each cycle is only whether it gets to issue.
The two pipelines are separate silicon. Everything DeepSeek does in the accumulator slide is bought with that fact.
What a GEMM kernel does
One CTA owns one output tile. K is 7168, so it walks along k in chunks.
The dashed box is the only part that repeats. Everything on the next three slides is a modification to something inside it.
Mixed precision framework
Change 1 of 3. Low-precision training had rarely been made to work at this scale. The compute-dense operations move to FP8; deliberately, not all of them.
The compute-dense kernels run in FP8. Figure 6 shows the representative case: the three GEMMs of a Linear layer, Fprop, Dgrad and Wgrad. That theoretically doubles compute speed, and it lets activations be cached in FP8 for the backward pass, which cuts memory as well.
Sensitive operators keep their precision. Embedding, output head, MoE gating, normalization and attention stay in BF16 or FP32.
So do the master copies. Master weights, gradients and optimizer states are held wide, because a running total in FP8 would round its own updates away.
FP8 E4M3 tops out at 448, with about two decimal digits. Nothing real fits in that, so every tensor has to be scaled before it can be cast.
Figure 6 from the paper, showing the Linear operator only. Every arrow into a GEMM is FP8; every arrow out of one is wider.
Fine-grained scaling
Change 2 of 3. The scale is set by the largest element in its group.
Activation outliers are per token and per channel, so the tile is narrow in both. Weights are static and better behaved, so a 128×128 block is enough, and it matches how the GEMM already tiles the work.
The tensor core accumulator
Change 3 of 3. The tensor core multiplies FP8 and accumulates internally. That accumulator is not FP32.
An undocumented hardware limit, found by measuring it. The fix is not in the tensor core. The accumulator width is fixed in silicon, so it is built around it.
DeepGEMM
FlashMLA for attention, DeepGEMM for the matmul. Neither appears in any paper.
