Lec 12 LASSO and Elastic Net Regularization Techniques

Main ideas and lessons

• The lecture builds on the previous topic of ridge regression (used to mitigate overfitting in linear regression) and introduces:
  • Lasso regression (Least Absolute Shrinkage and Selection Operator)
  • Elastic Net (a combination of ridge and lasso)

Overfitting mitigation

• Ridge reduces overfitting mainly by shrinking parameter magnitudes.
• Lasso addresses overfitting by both:
  • shrinking parameters, and
  • performing feature/parameter selection (driving some coefficients exactly to zero).

---

Lasso (Least Absolute Shrinkage and Selection Operator)

What “lasso” stands for / interpretation

• Least Absolute Shrinkage: encourages the “lowest” (small) parameter values.
• Selection: can make some parameters exactly zero, effectively selecting a subset of important features.

Lasso objective (loss function)

Lasso modifies the linear regression loss by changing the regularization term:

• Data fit term (same as standard linear regression):

  • [ \sum_i \left(y_i - \sum_j x_{ij}\beta_j\right)^2 ]

• Regularization term (lasso):

  • Instead of (\sum_j \beta_j^2) (ridge), it uses:
  • [ \sum_j |\beta_j| ]

So the overall lasso loss has the form:

• Squared error + (\lambda) × sum of absolute coefficients

Why lasso can set coefficients to exactly zero (geometric intuition)

• Consider two parameters, (\beta_1) and (\beta_2), and visualize:
  • Iso-contours of the loss (where loss is constant)
  • The feasible region imposed by the regularization

Ridge regression geometry

• Feasible region is inside a circle defined by:

  • [ \sum_{j=1}^{p} \beta_j^2 \le t ]

• The optimal point tends to occur where coefficients are unlikely to be exactly zero.

Lasso geometry

• Feasible region is inside a rotated square/diamond defined by:

  • [ \sum_{j=1}^{p} |\beta_j| \le t ]

• Because the corners/edges align with coordinate axes, the tangency/intersection point can occur at:

  • [ \beta_1 = 0 ]

  • (and similarly for other coefficients)

  • This is the mechanism behind feature selection.

Practical consequence: optimization method needed

• The absolute value penalty ( |\beta_j| ) is not differentiable at 0.
• Therefore:
  • Unlike ridge regression, lasso does not admit an analytic closed-form solution.
• As a result:
  • You must use numerical optimization.
  • The lecture suggests using a variant of gradient descent (because standard gradient-based methods need adjustment due to non-differentiability).

---

Choosing the regularization strength (\lambda)

• (\lambda) is a hyperparameter.
• It should be chosen using cross-validation so performance is good on:
  • the training set
  • and the test set (unseen data)

---

Elastic Net (combination of ridge + lasso)

Core concept

Elastic Net combines:

• ridge loss (squared coefficient penalty)
• lasso loss (absolute coefficient penalty)

Combined objective

The elastic net loss includes:

• Squared error term:

  • [ \sum_i \left(y_i - \sum_j x_{ij}\beta_j\right)^2 ]

• Plus two regularization parts:

  • Ridge part: (\sum_j \lambda_r \beta_j^2)
  • Lasso part: (\sum_j \lambda_l |\beta_j|)

(Equivalently, the lecture describes it as having both a squared and an absolute penalty.)

Model comparison strategy (important methodology)

Because it’s unclear which model will perform best:

• Evaluate and tune all of:
  • Ridge regression
  • Lasso
  • Elastic Net

Steps described:

1. Optimize each model’s hyperparameters (regularization parameters).
2. Compare performance on unseen test data.
3. Choose the model with good performance on both training and test sets.

• Metrics mentioned:
  • compute loss and (R^2) on train and test; prefer comparable and high performance.

---

Advantages and disadvantages (as stated)

• Ridge regression

  • ✅ Analytic solution
  • ✅ Helps remove singularity issues (e.g., from (X^T X))

• Lasso

  • ✅ Can set some coefficients exactly to 0 → feature selection
  • ❌ No analytical solution due to non-differentiability at 0
  • → must use numerical methods

• Elastic Net

  • ✅ Attempts to “combine the best of both worlds”
  • ❌ Still requires empirical comparison since the best method depends on the dataset

---

Speakers / sources featured

• Course lecturer / “I” (the main speaker delivering the lecture)
  • No other named speakers, sources, or citations are provided in the subtitles.