Deep Dive into LLMs like ChatGPT

High-level takeaway

The video walks through the full LLM pipeline (what happens “behind the text box”): pre‑training on web text, tokenization, neural‑network training (transformers), inference, post‑training (supervised fine‑tuning / “instruct” tuning), and reinforcement learning (RL / RLHF). It emphasizes how each stage changes what the model is and how you should use it, plus practical tips for prompt design, hallucination mitigation, tool use, and where to run models.

---

1) Pre‑training (data collection and preprocessing)

• Raw sources: massive web crawls (e.g., Common Crawl). Example dataset: FineWeb (Hugging Face).
• Typical filtering & cleaning pipeline:
  • URL/domain blocklists (spam, malware, adult content).
  • HTML text extraction.
  • Language filtering (e.g., keep pages >65% English).
  • Deduplication and PII removal.
• Result: large but manageable token corpora (FineWeb example: ~44 TB raw → ~15 trillion tokens after filtering).
• Goal: expose the model to diverse, high‑quality text so it can learn statistical token patterns.

2) Tokenization

• Conversion: text → bytes → subword/byte‑pair tokenization to trade sequence length vs. vocabulary size.
• Modern vocab sizes are large (example cited: GPT‑4 ≈ 100,277 tokens).
• Tools: tokenizer explorers let you inspect token IDs and tokenization behavior.
• Practical effect: tokens are the atomic units the model sees (not characters), which explains certain spelling/character‑level failure modes.

3) Model architecture & pre‑training optimization

• Architecture: Transformer (attention + MLP layers). Input is a token sequence; output is a probability distribution over the next token.
• Objective: next‑token prediction across windows (context length). Parameters updated by gradient descent to minimize loss.
• Scale: parameter counts range from millions to hundreds of billions (GPT‑2: 1.6B; modern models like LLaMA 3 are much larger).
• Compute: training uses many GPUs (H100s shown), multi‑node clusters, and large cloud nodes. Pre‑training is the most compute‑heavy stage (months on many GPUs).
• Reproduction example: the presenter reproduced GPT‑2 (reference: lm.c GitHub repo), noting historical cost estimates and that modern runs can be cheaper.

4) Inference behavior & stochasticity

• Inference = sampling next tokens from the model’s probability distribution (stochastic).
• Outputs are statistical “remixes” of training text—autocomplete‑style behavior, not exact retrieval (although memorization can occur for repeated documents).
• Base models behave like “internet text token simulators”; they are not yet assistants until post‑training is applied.

5) Post‑training / Supervised Fine‑Tuning (SFT, instruction tuning)

• Goal: convert a base model into an assistant that answers helpfully and safely.
• Data: curated conversational datasets where human labelers write prompts and ideal assistant responses according to instructions.
• Algorithm: SFT is the same next‑token prediction objective but applied to conversation‑structured data.
• Conversation encoding: system/user/assistant tokens and special turn markers are inserted; everything becomes a single 1‑D token sequence.
• Few‑shot / in‑context learning: include examples in the prompt to change behavior.
• Prompt engineering techniques: system messages, roleplay templates, and conversation scaffolding can make base models behave like assistants.

6) Hallucinations & factuality: causes and mitigations

Causes:

• The model is fundamentally a statistical token predictor and won’t “look things up” unless integrated with tools.
• Training distribution biases lead the model to answer confidently.
• Memorization of pervasive sources can produce verbatim regurgitation (e.g., Wikipedia).
• Internal uncertainty is not automatically verbalized.

Mitigations described:

• Teach the model to refuse when it doesn’t know: add training examples where correct output is “I don’t know.”
• Tool integration: enable the model to call external tools (web search, database, calculator, code interpreter). The model emits tool tokens, infra runs the tool, and results are fed back into context.
• Reward/penalize via supervised signals and RL (see RLHF).
• Engineering prompts: include source text in the context when precise answers are required (paste a chapter to summarize).

7) Tools & “working memory”

• Context window = the model’s working memory (tokens you supply). Data in context is directly accessible; network parameters act like vague long‑term memory.
• Example tools: web search (refresh memory), code interpreter (reliable arithmetic/string ops), system messages to set identity/behavior.
• Practical tip: when you need precise/novel facts, prefer providing them in context or ask the model to use tools rather than relying on internal memory.

8) Practical prompt & reasoning advice

• Distribute computation across tokens: ask for intermediate steps (chain‑of‑thought) rather than a single final token.
• For high numeric precision or counting, use tools (code execution) instead of relying on the model’s internal arithmetic.
• For exact recall, paste the source into the prompt so the model reads it in context.
• Use system messages to define persona/identity.
• For base models, use few‑shot prompts and conversation templates to simulate assistant behavior.

9) Cognitive/behavioral quirks & sharp edges

• Hallucinations: inventing people/facts from pattern matching.
• Memorization: ability to reproduce long passages (privacy/PII concerns).
• Tokenization artifacts: tasks requiring character‑level manipulation or fine counting can fail because the model operates on tokens.
• Limited per‑token compute: each forward pass has finite compute; complex reasoning must be spread over multiple tokens.
• Identity/self‑knowledge: the model has no persistent self; answers about its origin are statistical patterns unless explicitly programmed.
• Swiss‑cheese capability: broad competence but unpredictable failures on narrow edge cases.

10) Reinforcement learning & RL from human feedback (RLHF / RLFT)

Motivation:

• Let the model practice (trial & error) and discover token sequences that reliably produce desired behavior—analogous to doing practice problems after learning.

Process:

1. Generate many candidate responses (rollouts) for prompts.
2. Score responses:
   • For verifiable answers (math, code), use automatic scoring or LLM judges.
   • For subjective cases, collect human comparisons and train a reward model.
3. Train a reward model: fit a scoring network to human preference rankings.
4. Run RL (policy optimization) against the reward model to prefer higher‑reward responses.

Benefits:

• Emergence of more reliable chain‑of‑thought, longer solutions, backtracking, and improved reasoning patterns (example: DeepSeek R1).

Limitations / failure modes:

• Reward models are learned neural nets and can be adversarially exploited; long RL runs can “game” the reward and degenerate outputs.
• RLHF is sensitive to reward design and requires guardrails and human oversight.
• Historical analogy: AlphaGo’s RL discovered novel moves; similarly, RL can improve reasoning but is not a panacea.

11) Examples of models / releases / platforms referenced

• Papers and models: GPT‑2, InstructGPT, LLaMA / LLaMA 3, DeepSeek R1, AlphaGo.
• Hosted platforms / tools:
  • ChatGPT / OpenAI (assistant models, GPT‑4 family)
  • Hugging Face (datasets like FineWeb; inference playground)
  • Hyperbolic (hosting LLaMA 3 base)
  • Together.ai (model playground)
  • Lambda (GPU rental provider; H100 nodes)
  • LM Studio (local GUI for distilled models)
  • Tokenizer explorer tools (e.g., tiktoken/tokenizer demos)
• Leaderboards & news sources: LLM leaderboards, AI News newsletter, social media (X/Twitter).
• Reproducibility code: lm.c (GitHub) reproducing GPT‑2.

12) Practical how‑to / mini guides referenced

• Inspect tokenization: use a tokenizer explorer to view tokens/IDs.
• Reproduce a small GPT: pointers to GPT‑2 reproduction (lm.c) and modern cost differences.
• Renting compute: use cloud providers or GPU rentals (H100 nodes via providers like Lambda).
• Interact with base models: use Hyperbolic for LLaMA base; use few‑shot prompts and system messages.
• Convert a base model to an assistant (SFT): build a conversation dataset and fine‑tune on it.
• Hallucination mitigation:
  • Include “I don’t know” examples in training.
  • Integrate web search and code tools.
  • Use code for arithmetic/counting/string operations.
• Local experimentation: use distilled/smaller models and tools like LM Studio to run models locally.

13) Future directions & predictions

• Growing multimodality: joint training on text + audio + images via tokenization of other modalities.
• Agents & long‑running tasks: more agent frameworks that perform multi‑step tasks with human supervision.
• Greater tool integration: keyboard/mouse automation, web actions, and deeper tool ecosystems.
• Research needs: test‑time learning / parameter updates beyond in‑context learning, and methods for very long multimodal contexts.
• Deployment: models will become pervasive in apps but will still require human supervision and verification.

14) Key recommended practices for users

• Use LLMs as productivity tools, but verify outputs (especially facts, numbers, and code).
• Provide source text in context when precision matters; ask models to use tools for facts and arithmetic.
• Prompt design: request intermediate steps, chain‑of‑thought, or executable code for calculations.
• Be aware of tokenization quirks; use programmatic tools for exact string/counting tasks.
• Treat outputs as simulations of trained human labelers, not as infallible oracles.

Main speakers / sources cited

• Video narrator/presenter (creator of the video; unnamed in subtitles).
• Papers / organizations referenced: Hugging Face (FineWeb), OpenAI (GPT‑2, InstructGPT, ChatGPT / GPT‑4, RLHF work), Meta (LLaMA / LLaMA 3), DeepSeek (R1), DeepMind (AlphaGo), Anthropic, Google (Gemini).
• Platforms & tools: Hugging Face inference playground, Hyperbolic, Together.ai, Lambda, LM Studio, tokenizer web tools.
• Reproducibility repo: lm.c (GitHub).

Optional extras (from the summary)

• Produce a short checklist for building or evaluating an LLM/assistant (data, tokenization, compute, SFT, RLHF, tools).
• Extract the presenter’s specific step‑by‑step prompts for turning a base model into an assistant (template + system message).