RL for Agents Workshop - Deep Dive on Training Agents with RL and Open Source

Technological concepts & key points (agents + RL training deep dive)

1) Training agents, not just models

The talk frames modern progress as shifting from chat-based model outputs to agents that execute tasks semi-autonomously (e.g., code implementation, long-running tool use).

Examples referenced:

• ChatGPT: conversational interface for broad Q&A
• Cursor: code-oriented autocomplete/tab completion
• Claude Code (Anthropic): multi-agent / multi-step implementations with more autonomy

2) Task horizon is growing fast (benchmarks trend)

A central claim: task horizons are increasing quickly—tasks that previously took a skilled human about a day are increasingly achievable by frontier models in roughly half-day to day ranges.

Benchmark style mentioned:

• A curve from Meta using a human-relative success rate vs. time/hours trajectory, with improvement roughly doubling every 4–7 months.

Caveats:

• Benchmarks may be incomplete or “gameable,” but the overall trend appears strong.

3) Agents can handle long-running work; but RL credit assignment gets hard

As training shifts from RL “from verifiable rewards” (short rollouts, simple verifiers) to agent RL, the episode becomes:

• multi-step / multi-turn
• potentially multi-hour / multi-day (e.g., spinning up Docker, managing infrastructure)

New challenge: credit assignment

• With hundreds of tool calls and huge token counts, it becomes unclear which step caused success or failure.

Prior approach referenced:

• Process rewards (label intermediate steps), but described as not scalable enough.

4) Why train your own agent / model (cost, privacy, specialization)

Reasons discussed for training or customizing locally (even when proprietary models exist):

• Local deployment & privacy/personality
  • Avoid sending sensitive conversations to external labs.
• Serving cost / token economics
  • Frontier API pricing can change.
  • Cheaper models may be post-trained for specific products.
  • Examples of post-training for efficiency:
    • “Composer”-style model for Cursor
    • smaller/local models (e.g., from “the H company”)
    • Chroma search agent mentioned
• Domain specialization / out-of-distribution capability
  • Models can be “jagged” across domains; custom training can fill gaps.
  • Example: Dr. Tulu (Allen AI) using rubric-based RL to improve multi-step deep research-like retrieval/synthesis for smaller models (~7–8B).

5) Open-source RL training paradigm shift: RLVR → agent RL (inner loop + environment)

Earlier RLVR framing:

• Prompt → answer → stateless verifier → reward at end of rollout (often minutes)

Now with agents:

• Inner loop
  • Agent interacts with an environment
  • Environment provides observations (e.g., API calls/results)
  • Episode ends after many steps
  • Reward is computed after the trajectory completes, though sometimes dense via heuristics/rubrics

6) What an “environment” consists of (universal building blocks)

Environments were described as including:

• Tasks: what the agent should do
• Execution backend: sandbox, browser, bash terminal, code runner
• State management: DB/files/tool outputs and progress tracking
• Reward assignment: binary rewards or heuristics + rubrics (sometimes using other LLMs for dense feedback)

7) Ecosystem pieces for environments/harnesses

Multiple RL environment frameworks were listed (e.g., NVIDIA NeMo gym, Prim Intellect, Meta OpenM, and more).

Environment hubs emerging:

• Environment hub by Prim Intellect
• Open Rewards (General Reasoning)
• Hugging Face Spaces used as environments (for multiple framework types)

Goal: reduce friction so developers can plug environments into training/eval loops.

8) Training frameworks: move toward asynchronous RL

Modern paradigm: asynchronous training to prevent GPU “bubbles” when long rollouts stall optimization.

Key idea:

• Generators continuously produce rollouts
• Trainers update weights “in flight” once buffer staleness thresholds are met
• Policies can switch mid-generation (adds some off-policy behavior but improves throughput)

9) Evaluation (evals) are hard: saturation, overfitting, and reward hacking

Major evaluation issues discussed:

• Benchmark saturation
  • Public benchmarks saturate quickly, limiting signal for progress.
• Overfitting to public evals
  • Open models can underperform if benchmarks change, while proprietary labs may have richer internal evals/environments.
• Reward hacking in agent evals
  • Agent-enabled terminals/sandboxes increase the chance of exploiting benchmark weaknesses.

Recommendation takeaway:

• Build and run internal evals tied to your actual task, not only generic leaderboards.

10) “Scaffolds” / test-time compute strategies

Scaffolds were framed as structuring search/aggregation to improve performance:

• parallel best-of-n
• sequential refinement
• recursive self-aggregation

The claim: beyond static inference tricks, training with scaffolds can yield stronger performance.

Examples mentioned:

• Recursive self-aggregation used to improve performance on tasks involving Gemini
• Also referenced via OpenAI internal theoretical physics/generalization examples

---

Review/guides/tutorial-style contributions (benchmarks + eval talk + RLM + environments)

A) Benchmark/eval workshop segment: SWE-bench and beyond

Speaker: Ofir Press (SWE-bench / Artificial Analysis)

Core point: Benchmarks are “perishable” assets that saturate in months to a couple of years—so continuous creation is needed.

SWE-bench evolution described:

• Started with “school-exam”-like tasks (e.g., GSM8K)
• Moved to college-exam/general knowledge (e.g., MMLU)
• Then to human-authored programming tasks
• Now to real-world work / repo-level tasks (the SWE-bench paradigm)

Saturation stats mentioned:

• Anthropic “Methuselah” achieving 93.9 on SWE-bench verified (example of rapid improvement)

Proposed benchmark “stages”:

• Stage 4: full repo tasks (e.g., Commit Zero)
• Stage 5: tasks nobody has done before but are verifiable (compiler-scale tasks, “rewrite Linux kernel to go” style examples)

Recent benchmark examples highlighted:

1. Point
   • Physics question generation from professors
   • Verifiable numeric/equation answers via math equivalence
2. AlgoTuned (Algo Tune)
   • Iterative code optimization under a $1 per task budget
   • Must speed up existing algorithms while maintaining correctness
   • Verifies correctness via held-out tests; grades via speedup
   • Observation: cheaper models can sometimes do better due to more iteration opportunity
3. Code Clash
   • Two-agent coding competition in simulated arenas (e.g., Robot Rumble)
   • Agents iteratively improve scripts based on long logs over many rounds
   • Rankings via ELO
   • Emphasizes long-horizon difficulty not captured by SWE-bench

Benchmark design rules emphasized by Ofir:

• Maximize real-world usefulness (avoid “IQ-test”-style problems)
• Make benchmarks challenging (start low accuracy)
• Use deterministic verifiers rather than LLM-as-judge when possible
• Ensure difficulty is neither too easy (saturates) nor too hard (no improvement possible)

---

B) Recursive Language Models (RLMs): alternative way to get long-horizon reasoning

Speaker: Alex Wang (MIT / recursive language models)

Clarification: RLMs are not sub-agents and not merely context offloading to files managed by the model.

Core design:

• A thin wrapper around a base LM that provides access to a REPL-like environment (Python/Bash/IPython, etc.)
• The LM can call recursive language model/function calls as programmatic tools inside that REPL

Tool-calling distinction:

• Prefer programmatic tool calling where tool calls (including sub calls) are first-class primitives in the coding environment, rather than JSON-structured tool calls.

Motivation:

• Scaling long context directly to millions is hard
• RLMs enable reasoning beyond context windows by chaining calls where each call stays within a manageable context window

Training relevance:

• RLMs can be used in RL training to enable longer structured reasoning patterns

Scaffold-like behavior referenced:

• Related to “QED nano”-style ideas where models iteratively summarize intermediate reasoning traces to extend effective reasoning length

Benchmark referenced:

• Long-CoT (reasoning graphs requiring traversal), where properly trained RLM-style systems can excel

---

C) Environments-as-evals (Prime Intellect) + loop-based optimization

Speaker: Will Brown (Prime Intellect)

Central claim: evals and environments are essentially the same thing—a unified object encapsulating:

• tasks (dataset role)
• harness (agent/system interface role)
• metrics/rewards (reward function/rubric role)

“Disposable eval” framing:

• Evals should be ones you personally care about and iterate on, not necessarily permanent public artifacts.

Prime Intellect platform concepts:

• A verifiers library to build environments (tasks + harness + deterministic checks/rubrics)
• Harbor format referenced for packaging terminal-style agent harnesses
• Environments used for:
  • evaluation
  • optimization
  • prompt tuning
  • model selection
  • training runs

Example emphasized: TA2 (Taubench)-style simulated environments

• Telecom support benchmark using user simulation via LLM
• Focus on synthetic environments and transferring training from synthetic → real worlds

Continuous improvement loop described:

• Use system logging/traces from deployed agents to bootstrap new evals
• Replay prompts + accepted code diffs + criteria to build training signals
• Judge strategies include:
  • deterministic code-based checks
  • potentially LLM judges when questions are binary/near-verifiable and aggregated across multiple criteria

Practical outcome:

• Reduce friction so teams can generate evals/environments and use them to drive RL/SFT/prompt/model optimization.

---

Main speakers/sources (as mentioned in the subtitles)

• Lewis (opening talk; likely Lewis from the RL for Agents / Hugging Face context)
• Ofir Press (SWE-bench; evaluations)
• Alex Wang (Recursive Language Models; MIT / RLM paper)
• Will Brown (Prime Intellect; environments/evals platform)

Referenced sources/labs mentioned throughout:

• OpenAI, Anthropic
• Meta
• Mistral
• NVIDIA (NeMo gym)
• Prim Intellect (environments/harness framework)
• Artificial Analysis (SWE-bench tracking; benchmark context)
• Allen AI (Dr. Tulu example)
• DeepSeek (RLVR / verifiable reward framing; long CoT-like behavior)