AI Engineer Roadmap All lessons Interactive Quiz MernStackDev GitHub

Overfitting and Regularization

Intermediate · 10 min read · Updated · By Saurabh Pathak

In short: Overfitting is when a model memorises the quirks and noise of its training data and so does worse on new data; regularization is any trick that makes memorising harder, such as an L2 penalty, early stopping, dropout or simply more data.

Play the interactive lesson: Overfitting and regularizationRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Intermediate · Reading time: 10 min · Phase: 0 · ML Foundations

Before you start: AI, Machine Learning and Deep Learning (train, validation and test sets) and Loss and Gradient Descent.

Words to know:

Explain it like I'm new

Imagine fitting a curve through 10 dots. A straight line misses most dots: underfitting. A wiggly curve that passes through every single dot looks perfect on those dots, but between them it swings wildly. Draw a new dot from the same source and the curve is far off: overfitting. A smooth curve that follows the general trend is best.

As you make a model more flexible (more parameters, more training), two errors behave differently:

Ways to fight overfitting:

Watch it

Overfitting and regularization: Validation error is U-shaped; a small L2 penalty tames degree 9.
Overfitting and regularization: Validation error is U-shaped; a small L2 penalty tames degree 9.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

regularized loss = data loss + lambda * sum(w^2)        (L2 / ridge)
gradient step    : w <- w - lr * (dLoss/dw + 2 * lambda * w)     (weights "decay" a little every step)

lambda sets the trade-off: 0 means no penalty (free to overfit), very large means weights are forced toward 0 (underfit). Choose it using the validation set. Usually the intercept (bias) is not penalised.

Worked example. From ml_overfitting.py: data is y = sin(2πx) + noise (noise standard deviation 0.3, so no model can do better than an error of about 0.09). We fit a polynomial of each degree to just 10 training points and measure the mean squared error on those 10 points and on 300 separate validation points:

DegreeTraining MSEValidation MSE
10.30710.4509
20.30670.4529
30.11640.1592
40.08720.1694
50.07640.1580
60.07390.1670
70.06510.1976
80.06450.2110
90.00003.2875

Reading the table:

Now add L2 to the degree 9 model (the penalty is lambda * sum(w^2) added to the summed squared error, intercept not penalised):

lambdaTraining MSEValidation MSELargest abs(weight)
00.00003.28751.674e+06
1e-060.07690.158654.78
0.00010.08070.156122.94
0.0010.10120.178013.2
0.010.19070.27533.818
0.10.24020.35750.9933
10.29420.48640.3148

A tiny penalty (lambda = 1e-4) cuts validation error from 3.2875 to 0.1561, the best result in either table, and shrinks the largest weight from 1.674e+06 to 22.94. Training error rises a little (0.0000 to 0.0807): that is the point, we gave up a perfect fit to the training data to get a much better fit to new data. Push lambda too high and the model underfits again (validation 0.4864 at lambda = 1).

This is one random draw of 10 points, so the exact numbers vary with the data; the U shape and the rescue by regularization are the lasting lessons.

flowchart LR
    M[model too flexible] --> O[low train error, high validation error]
    O --> F{fix}
    F --> D[more data]
    F --> S[simpler model]
    F --> R[L2 / dropout]
    F --> E[early stopping]
    D --> V[check validation error]
    S --> V
    R --> V
    E --> V

Code example

From ml_overfitting.py (run python 00-ml-foundations/code/ml_overfitting.py):

def fit(xs, ys, degree, lam=0.0):
    """Minimise sum (X w - y)^2 + lam * sum_{k>=1} w_k^2 via the normal equations."""
    X = design(xs, degree)
    d = degree + 1
    A = [[sum(X[r][i] * X[r][j] for r in range(len(xs))) for j in range(d)] for i in range(d)]
    for i in range(1, d):
        A[i][i] += lam          # the L2 penalty: add lambda to the diagonal
    b = [sum(X[r][i] * ys[r] for r in range(len(xs))) for i in range(d)]
    return solve(A, b)

In real projects (needs pip install scikit-learn):

from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

model = make_pipeline(PolynomialFeatures(9), Ridge(alpha=1e-4))

In PyTorch, use torch.optim.AdamW(params, weight_decay=0.01) for weight decay, nn.Dropout(p=0.5) for dropout, and stop training when your validation loss has not improved for a set number of epochs (the "patience").

When to use it / when not to

SituationTry first
Train score much better than validation scoreMore data, then regularization or a simpler model
Both scores poorUnderfitting: a bigger or better model, better features, train longer; less regularization
Neural network, validation loss starts rising mid-trainingEarly stopping, dropout, weight decay
Very large datasetYou may need little regularization

Common mistakes

Recap

If you remember only three things:

Interview questions

1. How can you tell whether a model is overfitting or underfitting? <details><summary>Answer</summary> Compare training and validation error. A large gap (good on training, bad on validation) means overfitting. Both errors high means underfitting. A learning curve (errors versus training-set size or epochs) shows it clearly. </details>

2. What does L2 regularization do, and why does it help? <details><summary>Answer</summary> It adds lambda * sum(w^2) to the loss, so large weights are penalised and the model prefers smoother, simpler functions. In the repo's demo it shrank the largest weight of a degree 9 fit from about 1.7 million to 22.9 and cut validation error from 3.29 to 0.16. In gradient descent it acts as weight decay: each step multiplies the weights by slightly less than 1. </details>

3. How does dropout work, and what changes at test time? <details><summary>Answer</summary> During training each neuron's output is set to zero with probability p on every step, so the network cannot depend on any single neuron and behaves like an average of many thinner networks. At test time dropout is turned off and all neurons are used; in the common "inverted dropout" version the kept activations are scaled by 1/(1-p) during training so no extra scaling is needed at test time. </details>

Go deeper

Prev: Backpropagation · Next: Phase 1 · RAG Foundations · Back to phase: Phase 0