5 Training Part 2

Main ideas and lessons

• Loss landscapes explain how optimization methods affect model behavior

  • When designing an optimization method, you should consider how it changes the model’s behavior when viewed over a loss landscape (quality vs. model parameters).
  • Loss landscapes are common tools for analyzing optimization because they let you visualize/understand which regions of parameter space are good or bad.
  • However, a general theory is hard because landscapes change with:
    • different datasets,
    • different problems,
    • different model/hyperparameter representations.

• What determines the loss landscape shape

  • Model space (parameter space / hyperparameter space): the landscape is built over combinations of model parameters (e.g., in linear regression, parameters (a,b) define the space).
  • Objective function / loss function: determines how model quality is computed; changing the cost function (e.g., adding a regularization term) can dramatically change the landscape geometry.
  • Dataset / evaluation data: the loss is computed based on training data, so the loss landscape depends on the data distribution.
  • Regularization and knowledge-based terms can add new components to the loss and reshape the landscape.

• Optimization seeks local optima, but real-world goals are more complex

  • Peaks/valleys correspond to local optima (for maximization/minimization), and a local optimum can be characterized by gradient conditions (e.g., gradient (= 0) in all directions).
  • In practice (especially with huge neural nets), it’s infeasible to visualize or guarantee reaching a global optimum.
  • Therefore, most AI systems aim for better local optima rather than global or “true” optima.
  • The speaker distinguishes:
    • Local optimum: what the algorithm converges to.
    • Global optimum: best over the entire model space (usually intractable to guarantee).
    • True optimum: the optimum as data/information grows toward an ideal limit (e.g., training set size (\to \infty)); may not even lie within the chosen model space due to representational limitations.

• Why theoretical “optima” concepts matter for methodology

  • Even though “true/global optimum” are ambiguous and unobservable, they motivate how to assess and design methods:
    • whether the objective being optimized matches the desired real goal,
    • whether representational limits prevent the “true” solution from being expressible.

• Key optimization challenges in non-ideal landscapes

  • The speaker organizes landscape “conditions” and explains why gradient-based methods can fail:
    • Bad local minimum: convergence to a narrow region of low quality while a better global minimum exists elsewhere.
    • Oscillation: occurs when step size/learning rate is too large relative to landscape curvature; the method jumps past good regions repeatedly.
    • Flat landscape / indistinguishable states: gradients are tiny, learning is slow, and you can’t tell whether you’re truly optimized or just stuck in a flat/low-slope region.
  • In more complex real-world problems:
    • landscapes can be highly non-convex with many local optima,
    • escaping bad optima may require exploration.

• Exploration via randomness (and why it’s not fully understood)

  • Since landscapes are complex and changing, methods add randomness (e.g., via stochasticity) to escape poor local optima.
  • Stochastic Gradient Descent (SGD) is described as a practical way to add exploration:
    • Instead of full-batch gradients, it uses mini-batches so updates are noisy.
    • This noise can help jump between nearby basins of attraction.
    • Too much randomness can cause divergence.

• Learning rate / update scale is a central control knob

  • Learning rate (step size) determines how far parameters move each update.
  • Too small: slow convergence and inability to distinguish flat from genuinely optimal.
  • Too large: oscillation or divergence.
  • Therefore practical methods use schedulers/adaptive methods (e.g., mentioning Adam/additive or related schedulers) to control scale dynamically.

• Why practical deep learning uses initialization and normalization

  • Some real neural-network scenarios can produce zero gradients everywhere while still being low-accuracy solutions.
  • A practical mitigation is to initialize from pretrained models so the parameters start near more useful regions.
  • The speaker also links landscape behavior to network representation, and mentions that normalization (input normalization, batch normalization) and architectural/activation choices affect gradients and thus the landscape geometry.
  • Two broad approaches are mentioned:
    • simulate/structure landscapes (e.g., reduce unnecessary parameter space, encourage smoother/convex-like behavior),
    • reduce “wasted” regions so optimization is less likely to land in bad basins—though this requires knowledge/guarantees that are hard.

• Geometry mismatch and convergence speed

  • In higher-dimensional parameter spaces, landscapes may be skewed/elliptical rather than well-shaped (not circular).
  • If gradient-based step lengths don’t match the landscape curvature/scales in different directions, the method zig-zags, slowing convergence.
  • Practical fixes include balancing loss scales across components (if total loss is a weighted combination of terms) to make the landscape more well-conditioned and easier to traverse.

---

Methodology / instruction-like content (detailed bullet points)

How to reason about an optimization method using loss landscapes

1. Define the objective function (loss) that measures model quality.
2. Identify the model space
   • Determine the parameter/hyperparameter domain over which you build the landscape.
   • Recognize that representation choice changes the landscape.
3. Analyze how each factor reshapes the landscape
   • Dataset choice → changes loss evaluations → changes landscape.
   • Regularization / additional cost terms → can dramatically reshape the landscape.
   • Objective function design (including metrics and priors) → affects geometry (e.g., adding terms changes valley/peak structures).
4. Use landscape analysis to predict method behavior
   • Determine which landscape regions are “good” (low loss / high accuracy) and how your optimizer navigates them.
   • Ask whether the new method improves performance in the “good” regions and avoids “bad” regions.

Local vs global vs true optimum framing (what you should keep in mind)

• Expect convergence to local optima
  • Because reaching global optima is generally infeasible in large models.
• Evaluate internal quality repeatedly
  • The optimum induced by the objective may not correspond to the real desired solution.
• Consider “true optimum”
  • True optimum is conceptual: as data (\to \infty), the landscape and best solution may move.
  • True optimum may not exist in the chosen model space due to representational limitations.
  • Use these concepts to motivate what your method should aim for, even if you can’t precisely define/observe it.

Optimization design issues you must check when using gradient-based methods

• Bad local minima risk
  • Random initialization can land you in large basins of attraction for poor optima.
  • Recognize that statistical “success rate” claims may be misleading without landscape understanding.
• Oscillation diagnosis
  • If learning rate/step size is too large → parameters overshoot → cyclic behavior.
  • Remedy: reduce step size using schedulers/adaptive control.
• Flat-region stagnation
  • If gradients are tiny → progress is slow; loss changes may not tell you whether you’re truly optimized.
  • Remedy: adjust learning rate and/or use better initialization/exploration.
• Complex landscape navigation
  • As dimensionality and non-convexity grow:
    • escaping requires exploration/randomness,
    • too much randomness can diverge.

Gradient descent vs analytic solutions vs approximation (as described)

• Analytic approach (theoretical)
  • Unconstrained: set gradient (= 0) and solve differential/equation systems.
  • Constrained: use Lagrangian multiplier form (introduce (\lambda)) and solve resulting equations.
  • Practical note: analytic solutions become infeasible with large models/constraints; mainly used for theoretical foundation.
• Approximation approach
  • Use iterative gradient-based updates:
    • repeatedly move parameters in the direction indicated by the gradient,
    • repeat until convergence/sufficient progress.
  • The convergence depends on correct step-size control.

Stochastic gradient descent (SGD) exploration mechanism (as described)

• Start with standard gradient descent concept
  • Use gradient at current parameters to update once per iteration.
• Introduce mini-batch randomness
  • Instead of computing gradient on all data, compute gradients on sampled mini-batches.
• Interpretation
  • The noise acts as exploration:
    • may allow escape from poor local optima,
    • may also cause divergence if randomness is too strong.
• Practical dependency
  • The “proper amount” of randomness is unknown and must be controlled/tuned.

Practical tactics mentioned for deep learning optimization

• Pretraining / initialization
  • Initialize parameters near a region where useful gradients exist.
• Normalization and architectural choices
  • Use normalization (e.g., batch normalization) and other design choices to shape gradients and the effective landscape.
• Landscape shaping / space reduction
  • Reduce unnecessary parameter space or create smoother/convex-like geometry to avoid failure regions.
• Loss scaling / balancing
  • If using a weighted sum of losses, balance term scales to avoid skewed (elliptical) curvature that slows convergence.

---

Speakers / sources featured

• Speaker: Unspecified (single narrator; no named speakers appear in the subtitles).
• Sources: No external authors, papers, or institutions are explicitly named in the subtitles.