From GPT-2 to Kimi K3: How Modern LLMs Evolved

Main ideas, concepts, and lessons

• Scale matters, but architecture matters more than just “making it bigger.”

  • Kimi K3 is described as having ~2.8 trillion parameters versus ~124 million for the smallest GPT-2 size mentioned.
  • The video frames K3 as the result of many iterative “problems, trade-offs, and inventions,” not simply expanding an old model.

• GPT-2 fundamentals: how a language model predicts text

  • GPT-2 is introduced as a family of models using the Generative Pre-trained Transformer architecture:
    • Generative: produces text.
    • Pre-trained: learned before seeing the prompt.
    • Transformer: uses an attention-based architecture.
  • A parameter is an adjustable number learned from training data.
  • Model predictions work over tokens—pieces of text—assigning probabilities to the next token.

• Context-window limitation and why older clues need special mechanisms

  • The video compares prompts where the only difference is an early clue that falls outside the attention window.
  • If the clue is outside the window, the model shouldn’t condition future probabilities on it.
  • This motivates mechanisms that “carry” information forward.

• Evolution of memory mechanisms across architectures

  1. Recurrent networks (including LSTMs)
     • Maintain a running state (temporary memory) passed token-by-token through a relay/serial path.
     • LSTMs add gates to decide what to keep, update, or forget.
  2. Transformers (2017)
     • Replace serial recurrence with attention, enabling more direct paths to earlier tokens.
     • During generation, transformers still output one token at a time, but attention can directly reference earlier tokens.

• Tokenizer + embeddings: turning text into vectors

  • Raw text is too large/variable, so GPT-2 uses byte pair encoding (BPE):
    • Start from bytes, then merge common neighbors into reusable subword pieces.
  • Each token maps to:
    • an integer ID (an index/address, not inherently meaningful)
    • then an embedding vector (a learned vector representation)
  • The video emphasizes:
    • GPT-2 uses a fixed representation width (residual stream) of 768 numbers.
    • Learned “features” are distributed across many coordinates rather than one coordinate equaling one concept.

• Transformer block mechanics in GPT-2

  • GPT-2 processes tokens through 12 transformer blocks (for the described GPT-2 configuration).
  • Each block follows a repeated sequence:
    • Normalize
    • Attention
    • Add (residual connection)
    • Normalize
    • MLP (feed-forward)
    • Add (residual connection)
  • Attention details:
    • Computes Q (queries), K (keys), V (values) via learned linear projections.
    • Uses dot-product similarity between queries and keys, then scales by sqrt(head_dim).
    • Applies a causal mask so tokens can’t attend to the future.
    • Uses softmax to turn scores into attention weights.
    • Attention heads blend information from selected earlier positions.

• Output head and softmax sampling

  • After the final transformer layers, GPT-2 produces logits for each vocabulary token.
  • Softmax converts logits into next-token probabilities.
  • Generation samples a token and repeats.

• Training objective: next-token prediction and loss

  • Training shifts text so each position must predict the true next token.
  • Loss is the negative natural logarithm of the probability assigned to the true next token.
  • Intuition:
    • Low probability on the correct token increases loss sharply.
    • Improving an already good prediction changes loss less.
  • Learning follows the loop:
    • Forward → Loss → Backward (compute slope) → Step (update weights)

• Key-value (KV) cache and scaling context

  • During generation, GPT-2 can store past keys/values in a KV cache so new tokens reuse prior computation.
  • Trade-off: longer context means more cache data and higher cost.

• Shift toward long-context efficiency: linear attention variants

  • Linear attention avoids an ever-growing archive of position-by-position comparisons.
  • It keeps a fixed-size summary state (via matrix-like state updates).
  • Trade-off:
    • cheaper memory growth
    • but associations can “blur” because many moments share one compressed representation.

• Delta rule family: correcting a shared memory state

  • DeltaNet introduces an additive correction mechanism:
    • read the current state retrieval
    • compare to the desired value
    • write the difference (error signal) back into shared state
    • controlled by a learned strength beta
  • Parallel delta can reduce serial bottlenecks through chunking.
  • Gated delta adds learned decay alpha to control forgetting over time.
  • KDA (Kimi Delta Attention) improves forgetting control:
    • forgetting is channel-wise (different retention per state channel), not a single global decay.

• MLA and hybrid memory approach (Kimi linear)

  • Fully position-by-position attention (softmax attention) is expensive but preserves exact addressing.
  • Kimi’s Kimi linear combines:
    • KDA (efficient fixed-size blended state)
    • MLA (Multi-head Latent Attention) (latent codes act like cheaper per-position storage/indexing)
  • KDA provides ongoing summary memory; MLA periodically restores more precise retrieval.

• Kimi K3’s macro-architecture and repeated motif

  • K3 uses a repeated macrocycle:
    • three KDA layers + one gated MLA layer
  • The video states this rhythm repeats:
    • 23 times, then adds a final gated MLA layer
  • Layer counts given:
    • 69 KDA + 24 MLA = 93 attention layers
  • Each attention layer is paired with a feed-forward stage (functionally analogous to GPT-2 “blocks”).

• Mixture-of-Experts (MoE) for efficiency

  • K3 uses sparse expert routing in feed-forward layers:
    • experts are learned subnetworks
    • a router selects which experts run for each token
  • Video-stated routing:
    • two shared experts always run
    • router selects 16 out of 896 routed experts
  • Parameter stats:
    • ~2.8T total parameters
    • ~104B active parameters per token (weights used for that token; not temporary activations)

• Attention residuals: selecting useful depth, not just token positions

  • Like earlier models, K3 includes a residual highway that mixes information across layers.
  • Attention residuals allow preferring contributions from certain completed depth spans (e.g., around 12-layer boundaries).
  • This reduces cost while enabling reuse of the most relevant earlier “depth bundles.”

• Final synthesis: what changed from GPT-2 to K3

  • The original task is still next-token prediction.
  • What changes is how the model chooses:
    • which earlier positions to read
    • what memory to retain and how forgetting works
    • which experts to activate
    • which depth to reuse
  • The video frames K3’s huge parameter count as a map of these decisions, not just raw size.

---

Methodologies / instruction-like processes (detailed)

A) GPT-2 / transformer next-token generation (conceptual steps)

1. Convert prompt text into tokens using a tokenizer (BPE described).
2. For each generation step:
   • Provide the current context to the model within the context window.
   • Use transformer blocks to compute a representation for the final position.
   • Convert the representation to logits over the vocabulary via the output head.
   • Apply softmax to obtain next-token probabilities.
   • Sample or choose a token according to those probabilities.
   • Append the chosen token to the context.
3. Repeat until an end condition is met (not specified in subtitles).

B) GPT-2 training loop (gradient descent concept)

• For each training example:
  • Shift text by one token so each position has a true next token target.
  • Compute predictions for each position.
  • Compute loss as:
    • negative natural log of the probability assigned to the true next token
  • Average loss over usable positions.
  • Compute gradients of loss with respect to parameters.
  • Update parameters using an optimizer step (described as “gradient points downhill-ish”).

C) Delta rule / memory update (DeltaNet-like operation)

• For each update step:
  • Read what the current key retrieves from the shared recurrent state.
  • Compare retrieved value to the desired value.
  • Compute the difference (error signal).
  • Apply an additive correction to the state scaled by beta.
• Forgetting variant:
  • Apply gates that decay previous state by learned alpha, controlling retention vs replacement.

D) KDA (channel-wise gated forgetting concept)

• Maintain a recurrent state with multiple channels.
• For each token/head/channel:
  • use learned retention values to decide how strongly each channel persists,
  • instead of using one global forgetting value.

E) K3 hybrid memory schedule

• Use KDA layers to maintain efficient fixed-size recurrent summary state.
• Use MLA layers to periodically enable more precise position-wise retrieval using latent codes.
• Repeat the macrocycle to scale long-context processing.

F) MoE expert routing (per token)

• For each token at each MoE layer:
  • route to a subset of experts via a learned router.
  • always include two shared experts
  • select 16 experts out of 896 (as stated)
  • compute outputs only for the selected experts (sparse compute).

G) Attention residuals (depth selection concept)

• Instead of only mixing individual token positions, allow attention to select among completed layer groups (depth bundles), such as around 12-layer boundaries.
• Control how much each earlier “depth summary” contributes to the current representation.

---

Speakers / sources featured

• No explicit human speakers are identified in the subtitles.
• Model names / systems referenced:
  • GPT-2 (Generative Pre-trained Transformer)
  • LSTM / Long Short-Term Memory
  • DeltaNet / Delta rule concepts
  • KDA (Kimi Delta Attention)
  • MLA (Multi-head Latent Attention)
  • Kimi K3
  • Chinchilla (referenced as a comparison point: “Chinchilla 3”)