Video summary

Training Agents: Live tutorial on how to fine-tune a coding agent for continual learning

Main summary

Key takeaways

Technology

Summary (technological focus)

This live tutorial kicks off a multi-session series on “training agents”, with an emphasis on fine-tuning a coding agent for continual learning. The first episode demonstrates an end-to-end workflow for Supervised Fine-Tuning (SFT) on agent trace data, using an agent-driven orchestration approach—meaning an agent generates the training pipeline rather than the instructor writing every step manually.

Course/session plan and format

  • Planned cadence: likely every ~2 weeks (recorded; timing may adjust based on feedback).
  • At least 3 sessions initially:
    1. SFT on agentic traces (this session)
    2. Basic RL fine-tuning
    3. Advanced RL using environments/tools
  • Future sessions are expected to move toward long-running agent tasks and more complex environments.

Core technical workflow demonstrated (SFT episode)

1) Agent trace → dataset for training

  • The tutorial uses real agent traces from the Pie agent harness (author credited as Mario Zechner), rather than collecting new traces from scratch.
  • Example traces include:
    • Multi-turn tool calls
    • Branching “tree” conversation structures
  • Traces are described as instruction/completion pairs derived from sessions (e.g., Claude Opus 4.5 traces are mentioned). These can be adapted to other harnesses/models similarly.

2) Hugging Face trace visualization + orchestration

  • On Hugging Face, pushed traces in a “standard format” render as a readable visual trace.
  • The instructor starts by prompting an agent (example: “SFT train Gemma 4 2B model on this specific data set”) and supplies parameters to control:
    • the job
    • output artifacts

3) Hyperparameter sweep and job management (HF Jobs)

  • The agent orchestrates multiple HF Jobs to perform a parameter sweep.
  • It tracks each run via Trace/Track tool integration, including:
    • Tracking metrics across runs (subtitles mention “Track IO” / “Tracelo”)
    • Adding evaluation scores to the model README
    • Using a consistent project namespace for logging

4) Adapter + model publishing to Hugging Face Hub

The training process includes:

  • Pushing all adapters to Hugging Face repositories
  • Publishing the final selected weights/model
  • Producing a model-card-like table containing:
    • job IDs
    • dashboard links
    • repo links
    • evaluation links

5) Selection criterion: held-out eval loss proxy

  • For model selection, the agent uses held-out evaluation loss as a proxy for imitation quality.
  • This series segment uses no reward models and no RL yet—it is purely imitation learning via SFT.

6) Evaluation benchmarks and scoring

  • Evaluation uses Inspect AI, mentioning two benchmarks:
    • HumanEval
    • MBPP
  • The speakers note the evaluation approach may not be “best” yet and suggest improving eval design to reduce overfitting to trace-specific patterns.

“Under the hood” explanation: what the agent does in this SFT workflow

Instead of manually writing a full training pipeline, the agent:

  • Interprets the prompt as a real training request (not just planning).
  • Performs pre-flight checks (“orientation”) to avoid wasting HF Job spend, such as:
    • license/authorization checks (e.g., for model usage)
    • validating dataset existence/shape
    • verifying push permissions to repos
    • smoke tests / failure analysis (e.g., formatting errors or potential OOM risks)
  • Loads skills (modular instruction blocks) for:
    • TRL
    • Hugging Face Jobs
    • Track IO / Tracelo
    • HF authentication / CLI usage
  • Generates a verbose training script that uses TRL and configures an SFT trainer.

Training method: what “SFT on traces” means (key conceptual points)

Definition

  • SFT is continuation of pre-training, but trained specifically on instruction → completion data.
  • The tutorial reframes trace data as prompt/completion pairs, where the model learns to produce assistant/tool-call style outputs from the trace.

Masking and loss calculation (core mechanism)

  • Data is formatted using chat templates (via Transformers’ apply_chat_template).
  • Key mechanism: mask out the user portion so loss is computed only on the assistant completion.
    • Prompt tokens are set to an “ignore” value (e.g., -100 notation).
  • Result: the model learns to emulate the traced assistant behavior, including tool-use formatting.

What the tutorial intentionally does not do yet

  • No RL environments in this episode.
  • No reward functions yet.
  • The learning is strictly imitation learning (emulation of agent trace behavior).

Metrics tracked during training (Tracelo / Track IO dashboard)

The dashboard reports typical SFT trainer metrics, including:

  • loss (expected to decrease)
  • entropy (expected to trend downward)
  • token accuracy (expected to increase)
  • learning rate evolution (expected to decrease depending on the trainer schedule)
  • Train vs eval metrics, including an eval loss curve on a split (e.g., a “CatAway/DSR”-style split)

The instructor suggests improvements such as more training steps and more sweeps, and uses metric comparisons to monitor for signs of overfitting (e.g., similar train/eval trends).

Resources and reproducibility (as described)

  • A shared HF “trace called SFT on traces” reference is provided so viewers can inspect:
    • the agent’s tool calls
    • the skills used by the agent
    • training dataset links
  • The skills and the repo used as the agent “home directory” are linked in the video description.
  • The demo’s trained model is planned to be made public after the call.

Main speakers / sources

  • Ben (primary instructor; introduces the series and SFT explanation)
  • Sergio Paniego (developer advocacy; expands training workflow slides)
  • Quentin Galludec (maintainer of TRL; present in comments/behind the scenes)
  • Mario Zechner / Pie agent harness (source of the agent trace dataset used for SFT)

Original video