How to Earn More Reward

Summary

The lecture asks: How can a policy be improved to earn more reward without moving too far from a policy that already works? Its answer is to start with a trusted distribution of actions, sample possible outcomes, score them, and shift probability toward better outcomes. A temperature controls how large that shift can be.

1. Why imitation alone can fail

• Behavior cloning trains a driving policy to imitate human actions at logged states. It is evaluated open-loop: the policy is queried on recorded states, and its actions are compared with the human’s.
• But a vehicle acts closed-loop: each action changes the next state, which affects the next action. A small early error can take the vehicle into states missing from the demonstrations, where errors may grow.
• In a lane-keeping example, a policy trained on 40 expert demonstrations had low error on logged states, yet 32 of 100 closed-loop runs left the road. Adding 300 recovery demonstrations that covered larger offsets brought this to zero of 100, despite barely changing open-loop error.
• The lesson is that good imitation scores do not guarantee safe driving. Training must account for states the policy itself reaches.
• On 4,000 Waymo Open Motion test scenes, successive rounds of reinforcement learning raised the closed-loop score while the open-loop score stayed flat. The speaker notes that an earlier scorer was used, so the trend—not the absolute values—is the point.

The lecture contrasts three ways to obtain data from states outside the demonstrations:

1. Perturb recorded trajectories and label how to recover.
2. Let the policy drive, then train it back toward the logged trajectory.
3. Let it drive and learn from the outcome score.

The third approach—reinforcement learning—needs only a score for a rollout, rather than a correct action label at every state.

2. The shared optimization principle

The cloned policy induces a distribution over actions and, together with the vehicle dynamics and noise, over full rollouts. The goal is to improve expected reward while limiting departure from the cloned policy.

A convenient form is to minimize:

• Expected cost, plus
• A KL-divergence penalty from a base distribution, weighted by temperature (\lambda).

Equivalently, for reward rather than cost, the optimal distribution is the base distribution reweighted in proportion to the exponential of reward divided by temperature. A high temperature keeps the result near the base distribution; a low temperature concentrates it more strongly on high-reward outcomes.

This balances two risks:

• Optimizing the score alone can exploit gaps in the score and produce undesirable behavior.
• Staying too close to the original policy can prevent meaningful improvement.

3. The physics analogy: exponential weighting

The lecture uses physics to motivate this distributional solution:

• Light: In the wave account, every possible path to a mirror contributes an arrow. Most paths cancel through interference; paths near the shortest-time route reinforce one another. In the limiting classical picture, this yields the law of reflection.
• Imaginary-time version: Replacing rotating arrows with positive weights that decay exponentially with extra path length makes longer paths contribute less. This positive-weight form is the analogy used for control and learning. The lecture cautions that real-time light and matter involve complex amplitudes; the KL-style positive weighting applies to the imaginary-time analogy.
• Classical and quantum mechanics: Classical motion follows a path where action is stationary. Quantum mechanics assigns amplitudes to many paths, with the Planck constant governing how quickly their phases vary.
• Statistical physics: Gas molecules under gravity follow a Boltzmann distribution. Low temperature concentrates probability at low energy; high temperature spreads it out. More generally, minimizing expected cost plus temperature-weighted KL divergence from a base distribution gives an exponentially reweighted distribution. This is the Gibbs variational principle.

Across these examples, the same pattern appears: a base distribution is tilted toward lower cost (or higher reward), and temperature controls how concentrated the result becomes.

4. Sampling-based control: MPPI

The lecture applies the principle to a vehicle steering around an obstacle. The reward includes collisions, leaving the road, clearance, target deviation, and a smaller steering-effort penalty. Because many terms are discontinuous, gradient descent is unsuitable; exhaustive search is too large, so the controller samples candidate plans.

Three methods are compared:

• Random shooting: Sample plans and execute the highest-reward one. It is simple and parallelizable, but depends heavily on chance and discards information from all other samples.
• Cross-entropy method (CEM): Fit a Gaussian distribution to the top-ranked “elite” samples and repeat. It uses more samples, but its hard cutoff gives full weight to one sample and none to a nearly tied one.
• MPPI: Assign every sample a smooth Gibbs weight based on its cost, then use the weighted mean of the samples. Temperature moves the method between near-uniform averaging and near-selection of the best sample.

In the example, plans comprise 20 yaw-rate commands over slightly more than three seconds. The practical MPPI loop is:

1. Sample a batch of candidate paths from the current state.
2. Roll out the paths and calculate each path’s cost.
3. Convert costs to Gibbs weights, giving more weight to lower-cost paths.
4. Compute the weighted mean of the first control steps.
5. Apply that first step, observe the new state, and replan.

This is a form of model predictive control. The derivation connects the control-effort cost to a KL divergence between controlled paths and paths generated by noise alone. In the stated problem, the temperature corresponds to the noise parameter. A temperature that is too high makes the control average toward zero; one that is too low lets a few noisy samples dominate. Performance is best near the value indicated by the derivation.

5. Learning a policy with policy gradients

MPPI can repeatedly plan online, but it requires many rollouts each time. A network can instead learn in advance and produce an action in one evaluation. Its starting point is the cloned policy; the training target is the same reward-weighted distribution, with a KL penalty to keep the learned policy close to the clone.

The lecture illustrates this with a choice among 35 action tokens, each specifying a jerk and steering rate held for three seconds. The cloned policy assigns 62% of its probability to actions that collide or leave the road. The reward is discontinuous, but policy gradients can learn from sampled outcomes without differentiating the reward or simulator:

• REINFORCE: Increase the likelihood of sampled actions in proportion to their reward.
• Baseline: Subtract a reference reward, such as average reward, so better-than-average actions are reinforced and worse-than-average actions suppressed. Subtracting a constant preserves the expected update while reducing variance.
• GRPO: Compare several rollouts from the same scene, normalizing each outcome by its group’s mean and standard deviation. This helps compare outcomes within a scene and reduces dependence on reward scale. GRPO also adds a KL penalty toward the cloned policy.

In a toy task of steering through a gap, REINFORCE learned but continued to update after success and its success rate fluctuated. A running-average baseline settled at 100% when all samples succeeded. GRPO converged fastest in the example, though groups in which every sample tied provided no learning signal.

Temperature determines how strongly the learned policy departs from the clone. The lecture emphasizes that the score itself may be imperfect: if it misses important failures, it should be trusted less, which calls for a larger temperature.

6. How MPPI and policy gradients relate

For a Gaussian policy with fixed variance, MPPI’s weighted update to the action mean can be expressed as a weighted sum of gradients of log probability. When temperature is large relative to reward variation, the exponential weights are approximately linear in reward, yielding a REINFORCE-like update with a baseline.

Thus, the lecture presents policy gradient as a small-step limit of MPPI:

• MPPI reweights samples to form a new action distribution directly.
• Policy gradient makes a gradual update to network parameters, storing the improvement so the network can generalize to other states.
• KL penalties and clipped updates help keep gradient steps within the region where the approximation is useful.

7. Application to driving

The described planner is first trained by imitation on human driving logs. It then drives in a simulator, receives closed-loop scores, and is fine-tuned with reinforcement learning:

1. Collect several rollouts per scene from the current planner.
2. Score them with a closed-loop metric combining collisions, drivable-area compliance, progress, comfort, and other checks.
3. Compare rollouts from the same starting state; reinforce those above the group average and suppress those below it.
4. Apply a KL penalty to keep the planner near the cloned policy.
5. Repeat from the updated planner so training covers states it actually reaches.

The speaker says the simulator was checked step by step against official nuPlan code and scoring. This matters because a mismatch could teach the planner to exploit the simulator rather than drive well.

One example shows a cloned planner hitting a pedestrian at 12.4 seconds, while its reinforcement-learned version passes without contact. The lecture cautions that one scene is not sufficient evidence; evaluation is performed over the full test set.

Four broad uses of reinforcement learning in driving are identified:

1. Train from scratch in a fast simulator, without human data; this can require billions of steps and does not by itself ensure human-like behavior.
2. Fine-tune an imitation-trained planner using closed-loop scores—the approach described in the lecture.
3. Post-train large vision-language-action planners, which may receive fewer rollouts because they are large and expensive to evaluate.
4. Teacher–student training: Train a teacher with reinforcement learning in a cheaper simulator, then train a camera-based student to imitate it.

The lecture also describes a camera-only challenge setting in which the policy receives camera images and a route, but no map or object list. Rendered images reflect the vehicle’s actual position, so the policy must respond to its own drift. The speaker reports examples where a submitted model improved over an earlier version and says the team ranked first among 50 teams on the public leaderboard at the time. These are presented as claims made during the lecture; the challenge was still running.

Main takeaway

Across physics, sampling-based control, and reinforcement learning, the procedure is the same:

1. Start from a distribution you trust.
2. Sample possible outcomes.
3. Score those outcomes.
4. Shift probability toward better ones, using temperature or a KL penalty to limit the shift.
5. Repeat from the updated distribution.

Speakers and sources featured

• Speaker: One unnamed lecturer; no additional named speakers are identifiable in the subtitles.
• Human driving logs and demonstrators: The source of behavior-cloning data and recovery demonstrations.
• Waymo Open Motion Dataset: Cited for evaluation on 4,000 test scenes.
• nuPlan simulator/code and scoring: Cited in connection with simulator verification and closed-loop planner evaluation; the subtitles render the name as “new plan” in places.
• Driving simulator and reconstructed challenge scenes: Used for rollouts, scoring, and camera-rendered evaluation.
• Physics concepts discussed: Reflection and wave interference, classical and quantum mechanics, and the Boltzmann distribution/Gibbs variational principle.