How LLMs Actually Generate Text (Every Dev Should Know This)

How LLMs Generate Text (from Prompt to Output)

The video explains that chat-style LLMs (e.g., ChatGPT, Claude) don’t “know” the full sentence before generating. Instead, they build the response one token at a time as probabilistic guesses from a large vocabulary (over 100,000 token options, depending on the model).

It outlines five steps that occur each time you send a prompt and the model generates every next token:

1) Tokenization (text → tokens)

• LLMs don’t read words directly; they read tokens.
• Tokenizers split text into efficient subword units:
  • Common words may map to one token (e.g., “the”).
  • Uncommon/long words can become multiple subword tokens.
• Token IDs are then used downstream.

Practical impact: API “max tokens” limits are not word counts (roughly ~3,000 English words for ~4,960 tokens in the example).

2) Embeddings (tokens → vectors with meaning)

• Each token ID is converted into a high-dimensional vector (thousands of dimensions).
• Example claim: GPT-3 uses 12,000+ numbers per token.
• The embedding space captures relationships learned from context:
  • “king” is close to “queen”
  • “Python (language)” is near “JavaScript”
  • Different senses (e.g., “Python” the snake) can be far apart.

• Includes a classic vector arithmetic example: king − man + woman ≈ queen

• Programming-related terms form clusters in embedding space (e.g., functions/methods/variables separate from database/SQL terms).

3) Transformer + Attention (context processing)

• Embeddings feed into a transformer network with many parameters (billions).
• The key mechanism emphasized is attention:
  • For each token, the model decides which other tokens to focus on.
  • Example: in “The cat sat on the mat because it was tired,” the pronoun “it” refers to “cat”, not “mat”.
• Attention runs through multiple attention heads in parallel to capture different relationships.
• Model stacking mentioned:
  • GPT-3: 96 layers
  • Llama 3 70B: 80 layers

Output of this step: contextualized vectors encoding token meaning and relationships across the whole input.

4) Probabilities (transformer → scores over the vocabulary)

• After processing context, the final layer scores every possible next token in the vocabulary.
• Example vocabulary size mentioned: Llama 3 ~128,000 tokens.
• Scores (called logits) are converted to probabilities using softmax (so probabilities sum to 1).
• The model doesn’t “choose the correct sentence”—it samples a path through this huge probability space.

5) Sampling + looping (choose next token, repeat)

Generation proceeds by:

• Picking one token according to a decoding strategy.
• Appending it to the input.
• Repeating until:
  • an end-of-sequence token appears, or
  • a length limit is reached.

Common strategies described:

• Greedy decoding: always pick the highest-probability token
  • Pros: consistent
  • Cons: often boring
• Temperature:
  • Low temperature (e.g., 0.2–0.4) → sharper distribution → more predictable, safer outputs (recommended for precision/coding)
  • Higher temperature (e.g., ~1.0+) → flatter distribution → more varied choices but can reduce coherence (too high can become incoherent)
• Top-p / nucleus sampling:
  • Sample only from the smallest set of tokens whose cumulative probability reaches P (e.g., 0.9)
  • The number of candidate tokens can vary depending on confidence

Why latency and “predicting the script” aren’t straightforward:

• Longer outputs slow down because each new token requires attention over all previous tokens.
• The token at position N isn’t predetermined—it’s decided only when that step is reached.

---

Three Actionable Insights Given in the Video

1. Hallucinations aren’t “lying”

   • The model generates plausible, pattern-matching text from probabilities—even if facts are wrong.
   • Recommendation: verify factual claims, especially when outputs sound confident.

2. Temperature doesn’t “make creativity real,” it changes randomness

   • Higher temperature increases selection of lower-probability tokens.
   • Recommendation: use low temperature for deterministic tasks (coding, extraction, formatting).

3. Context limits reflect compute/architecture

   • Attention has quadratic complexity with sequence length (every token attends to every other token).
   • Recommendation: context limits are a computational constraint, not merely a product restriction.

---

Main Speakers / Sources (as Referenced in the Transcript)

• OpenAI’s tokenizer is shown as an example (no specific person named).
• The explanation references model families: GPT-3 and Llama 3 (no specific narrator named in the subtitles).