Training Agents 4: From reward functions to environments.

Summary (technological concepts + features/experiments)

1) Why RL “environments” matter for agent training

• The session explains reinforcement-learning (RL) environments as the mechanism that provides stateful dynamics for agents: agents take a sequence of actions and receive observations reflecting the environment’s current state.
• Key motivation: unlike earlier setups where rewards could be produced from static traces, RL training requires interactive rollouts (“act → environment updates → observe/reward → act …”).

2) How environments fit with TRL (no new RL algorithm)

• The talk emphasizes that they don’t introduce new TRL training algorithms; instead they build on existing TRL components, specifically GPO/GRPO trainer (noted as the same trainer used earlier).
• In the GRPO/GPO setting:
  • If every rollout in a group receives the same reward, the advantage becomes zero, so the model cannot learn.
  • Therefore, environments (or reward design) must yield reward variability across rollouts.

3) Environment definition and components (formal structure)

• Simple definition: an agent acts in an environment and gets back an observation of the environment state.
• In practice, an environment includes:
  • Task: what the agent must accomplish (e.g., fix a GitHub PR, generate a Blender representation).
  • Runtime: compute + software dependencies (e.g., GPU needs, packages like Blender, repo code state).
  • Grading / Verifiers: checks that compute rewards/rubrics from the outcome/observation.
• They discuss a separation for safety and anti-cheating:
  • Sandbox (task execution): runs the agent’s tool/code safely.
  • Verifier/reward computation: isolated from the agent so the agent can’t learn to hack reward signals or infer hidden logic.

4) Examples of turning tasks into environments

• Chess: board = environment “world”; pieces’ positions = state; moves = actions; scoring = reward.
• GitHub PR (“repo to RL” style):
  • Issue/task becomes what the agent must do.
  • Code state (branch/commit) becomes environment state.
  • Tests from the completed PR become the grading signal used to compute reward.

5) “Capability cycle” ecosystem framing

They propose an ecosystem loop for capabilities:

1. Agent discovery (find a capability in a model/harness)
2. Benchmarking (verifiers/judges evaluate capability)
3. Data/environment creation (environment mirrors benchmark tasks)
4. Training loop (train using environment; deploy; iterate harness/environment as models improve)

Environments are described as the “medium” for scaling tasks that are otherwise difficult under static evaluation.

---

OpenM: product/library features for defining and deploying environments

6) OpenM environment interface (API + workflow)

• Environments are implemented using an OpenM-style class with core methods like:
  • reset(): start a new episode/world state
  • step(action): advance the world via the agent’s action
  • returns observation + reward + done status
• Intended loop: policy/model interacts with environment via reset/step to produce training rollouts.

7) Deployment + sharing

• OpenM environments are self-contained applications deployable as Docker containers.
• Can run on multiple backends (examples mentioned):
  • Hugging Face sandboxes/spaces
  • Modal
  • Daytona
  • Local machines, Kubernetes, etc.
• Environments can be:
  • Shared/imported from Hugging Face Hub
  • Forked to create variants

8) CLI-driven automation (agent-assisted environment building)

• An OpenM CLI supports:
  • generating boilerplate environments
  • importing environments from other libraries (e.g., verifiers/reward tooling)
  • pushing to the hub
  • pulling/forking existing environments
• Upcoming features (as stated):
  • discover: agent searches hub for relevant environments by domain (e.g., “3D representations”)
  • validate: agent checks environment quality for training suitability

---

Practical experiments: coding agents with TRL + environments

9) Experiment type A: simple Python-coding environment + GRPO

• Setup:
  • Dataset of Python problems (example dataset referenced as “most basic Python problems”).
  • Environment exposes a run python tool to execute generated code inside an isolated place.
  • Reward derived from whether tests pass.
• Training loop:
  • Generate rollouts (e.g., 4 at a time).
  • Score via environment reward.
  • Update policy using GRPO.
• Observations reported:
  • Training reward fluctuated (bouncing up/down) during learning on mixed difficulty tasks.
  • Evaluation reward improved steadily, indicating general learning despite noisy training curves.
• Practical notes:
  • Runs in HF environments/instances; described as fast (example runtime ~30 minutes mentioned).
  • Highlights training a small model (example: 1.7B parameters) to learn coding behavior quickly.

10) Experiment type B: longer-horizon coding harness (OpenCode) with captured proxy

• Motivation: some harnesses have complex tool-calling loops that don’t fit the “trainer owns the loop” assumption.
• They introduce a mechanism (described as a captured proxy in the middle):
  • Harness executes many tool calls / potentially spawns agents.
  • Proxy captures the full interaction trace and sends it back so TRL can optimize the policy.
• Isolation + scaling:
  • Environments run on HF sandboxes (more scalable than HF spaces for parallel runs).
  • Uses Docker images with the coding harness stack (example: OpenCode).
• Reward:
  • fraction of hidden test cases passed.
• Model/training choices mentioned:
  • A larger model variant used (example: 3.8B parameters).
  • Uses SPO+ harness/worker (instead of only the simpler GRPO loop mentioned earlier).

11) Multi-harness extensibility

• They suggest this extends to multiple coding harnesses (examples listed: OpenCode, CodeX/OpenCode variants mentioned, etc.), with close-to-ready integration between:
  • TRL (training loop)
  • OpenM (environment deployment)
  • a harness-level rollout system

---

Reward hacking / evaluation guidance (analysis question)

• They address long-horizon agent training:
  • No “silver bullet” to distinguish genuine improvements from reward hacking.
• Recommended approach:
  • create rubrics/metrics that measure logical behavior (not just the reward being optimized)
  • use sanity checks/trackers dashboards where metrics should increase if behavior is genuinely correct
• Example warning signs:
  • repeated tool usage patterns that look like exploitation (e.g., consistently calling the same tool)
  • lack of alignment between task-logic metrics and reward improvements

---

Training recipes and framework questions (Q&A highlights)

• Environments can be used for multiple post-training steps:
  • Evaluation only (no training)
  • RL with GRPO/GPO-like methods
  • Distillation using environment-generated trajectories (teacher/student judgments)
  • SFT using traces produced inside an environment
• Reward density:
  • Reward can be per tool call/step or for the complete task, depending on what signals exist.
  • For learning to happen in grouped methods, there must be non-uniform rewards across rollouts; otherwise the advantage signal collapses.
• Framework bias:
  • TRL is recommended as a starting point for post-training/reward-based workflows.
  • Mentions alternative tooling/libraries (e.g., unsloth-focused workflows, production RL libraries).

---

Main speakers/sources

• Sergio (main collaborator/experiment presenter; referenced as replying in comments and running practical demos)
• Ben (asked/posed at least one key question about reward hacking)
• Lead speaker (sets theory/architecture; referenced as speaking before Sergio takes over)

Sources/projects mentioned:

• repo-to-RL (project by Adita Kavi)
• OpenM (library described; environments + CLI)
• OpenCode / coding harness examples
• Deepc paper (environment generation/validation pipeline referenced)