Anthropic Researcher explains how to train LLMs

Summary

Anthropic researcher Jeremy Berman gives an intuitive overview of how modern language models are built, from neural-network fundamentals through pre-training, post-training, and reinforcement learning. He presents the process as a progression: a model first learns patterns in data, then is trained to act like an assistant, and can later be optimized to reason through tasks using trial and error.

1. Neural networks: learning a function by adjusting parameters

• A neural network can be understood as a function that maps inputs to outputs. In the example, the inputs are an apartment’s location, size, and number of bathrooms; the output is its price.
• A simple model combines its inputs using adjustable numerical values, described as “knobs” or weights. These weights determine how much each input influences the result.
• More layers and parameters allow the model to represent more complicated relationships between inputs and outputs.
• The model’s loss measures how far its predictions are from the correct answers. Training aims to reduce that loss.

2. How training adjusts the “knobs”

The speakers explain the basic training loop:

• Give the model inputs and known correct outputs.
• Have it make predictions and calculate how wrong they are.
• Use automatic differentiation tools—such as PyTorch—to calculate how changing each parameter would affect the loss.
• Adjust the parameters in the direction that should reduce the loss. The learning rate controls the size of each adjustment.
• Repeat this process over many training steps.

This directed adjustment is called backpropagation. The explanation treats frameworks such as PyTorch as tools that track the calculations and help compute how each parameter contributed to the error. The apartment example is explicitly a toy demonstration, not a production training recipe.

3. What a language model learns

• A language model is a type of neural network, but its inputs and outputs are text represented as tokens—pieces of words or groups of words.
• Rather than directly outputting a single number, it predicts probabilities for possible next tokens. Training makes the correct next token more likely.
• In broad terms, the model learns patterns in the text it was trained on. Given a prompt, it generates a continuation that seems likely according to those patterns.
• Language gives the model many more possible relationships and outputs than a narrow numerical task, so it generally requires more data, parameters, and compute to learn a task to a comparable level.

4. Pre-training: learning to autocomplete

• Pre-training teaches a model to predict and continue text from a large collection of examples.
• A pre-trained “base model” is therefore primarily an autocomplete system. It may imitate styles or produce plausible text, but that does not guarantee it will be helpful, coherent, or true.
• The demonstration with a Qwen model illustrates that its completions reflect patterns in its training data, including code and other text.
• The host notes that a model can only reproduce information it has encountered in some form; Jeremy agrees, while clarifying that exposure to a person’s material would not guarantee verbatim or accurate quotations.

5. Post-training: turning a base model into an assistant

The speakers describe post-training as a way to teach the base model how to behave in a conversational role:

• Format examples as exchanges between a user and an assistant.
• Train the model on examples of desired assistant responses, which may initially be written by people or generated as training data.
• Include a signal indicating when the assistant’s reply is finished, so the model learns when to stop.
• The model learns the patterns of assistant-style responses and can then be prompted in that format.

This process is presented as a way to convert a general text-completion model into an instruction-following assistant. The speakers note that simply learning to sound like an assistant does not ensure accuracy: a model may still confidently make mistakes.

6. Prompting for step-by-step problem solving

• The video demonstrates that prompting a model to “think step by step,” or prompting it to begin with that phrase, can improve its performance on some tasks.
• Jeremy distinguishes this prompting technique from reinforcement learning: a prompt can encourage a model to produce a reasoning-style answer, while reinforcement learning is a training method intended to improve the model’s problem-solving behavior.
• The speakers discuss OpenAI’s o1 as an early public example of a model designed to reason more effectively, and DeepSeek R1 as a later example that helped make reinforcement-learning methods more visible.

7. Reinforcement learning: improving through attempts and feedback

Jeremy explains a simplified version of reinforcement learning for tasks such as math problems:

• Have the model attempt the same problem multiple times, often while producing a step-by-step solution.
• Score each attempt—for example, by checking whether it follows the desired format and reaches the correct answer.
• Compare each attempt’s score with the average score across the group. This relative measure is called an advantage.
• Make tokens in better-than-average responses more likely and tokens in worse-than-average responses less likely.
• Repeat the process so that the model gradually becomes more likely to produce successful solutions.

A simpler alternative, called rejection sampling in the discussion, keeps successful attempts and trains on them while discarding failures. Jeremy says this approach does not learn directly from negative examples and can reinforce irrelevant details. He describes GRPO—group relative policy optimization—as a method that uses comparisons among multiple attempts to learn from both stronger and weaker results, while reducing some of the noise.

This kind of reinforcement learning is described as on-policy because the model generates the answers used for its own training. The model is adjusted to make its own successful attempts more likely. Jeremy says this can be less disruptive than training on someone else’s reasoning, although training can still degrade other capabilities.

8. Why pre-training and post-training are often separated

• Pre-training learns from broad data, such as text from the world, to build a general next-token predictor.
• Post-training shapes the model toward a particular role, such as being a helpful assistant.
• Jeremy cautions that continuing to train a post-trained assistant on data from a different distribution—for example, a code corpus—could make it imitate that data and weaken its assistant behavior.
• He notes that researchers are exploring ways to integrate reinforcement learning more broadly into training, so the separation may not be permanent.

9. Can models discover new things?

• The video addresses the criticism that a model trained to predict likely text can only repeat what it has already seen.
• Jeremy argues that reasoning can build new abstractions and combine known information in ways that lead to conclusions not explicitly present in the training examples.
• He suggests that training models to reason over longer horizons could help them make new discoveries, while acknowledging that the extent of this ability remains to be seen.
• The discussion mentions that models are beginning to make progress on some unsolved mathematical problems, but frames future discovery as a possibility rather than a settled outcome.

10. Learning the field

Jeremy’s career advice is that AI research is learnable without a PhD. He recommends following one’s intuition, choosing topics that spark curiosity, and pursuing them deeply. The host adds that learners can experiment with open-weight models and free tools, and use coding assistants to generate or explain code.

Speakers and sources featured

Speakers

• Matthew Berman — Host and interviewer.
• Jeremy Berman — Anthropic researcher and guest.

Technical tools, models, and methods mentioned

• PyTorch, JAX, and TensorFlow — Automatic differentiation and machine-learning frameworks.
• Hugging Face — Mentioned as a place to obtain model files.
• Qwen — The open-weight model used in the demonstrations.
• ChatGPT and Claude — Mentioned as examples of modern assistants.
• OpenAI o1 — Mentioned in connection with public reasoning-focused models.
• DeepSeek R1 — Mentioned in connection with reinforcement learning for reasoning.
• Backpropagation, rejection sampling, and GRPO — Training methods discussed.