1.4.3 — Overfitting and Underfitting

1. Two errors, not one

The training set shows how closely a model fits the examples used to estimate its parameters. A validation set helps assess whether the learned pattern transfers to data that did not update those parameters. Compare the same metric or loss under comparable conditions.

Low training error alone is not enough. We need to observe both training and validation behaviour. Keep the test set for a final independent evaluation; do not repeatedly use it to choose a model.

2. Underfitting: the model misses the pattern

Underfitting occurs when a model is too simple for the task, its features are insufficient, constraints are too strong, or optimization has not found suitable parameters. The model already makes substantial errors on training examples, and validation performance is usually not much better.

A typical signal is high training loss together with high validation loss. Possible actions include checking features and task formulation, choosing a more expressive model, reducing excessive regularization, or verifying convergence. Training longer is not always the answer: first identify the actual constraint.

3. Adequate fit: the pattern transfers

With useful generalization, training error is reasonably low and validation error remains close without substantial deterioration. A small difference is natural because the two sets contain different examples and sampling variation. What matters is stability and whether the metric meets the task requirements.

Validation loss does not have to be lower than training loss. Regularization, dropout and related methods may deliberately make training harder, so training loss can sometimes be higher. Interpret the values in the context of the algorithm and metric.

4. Overfitting: the model adapts to the sample

Overfitting occurs when a model reproduces training examples too closely, including noise and sample-specific details. Training error keeps decreasing, but validation error stops improving or rises. The model describes familiar examples well but transfers the pattern less reliably.

Causes may combine: excessive model capacity, too little diverse data, noisy labels, information leakage, too many tuning decisions, or repeatedly optimizing against one validation set. A large train–validation gap is a useful signal, not proof by itself: check the split, distributions and measurement.

5. Learning curves and early stopping

During training, training loss often continues to decrease. Validation loss may first decrease, then stop improving and rise. This is a common overfitting pattern. Early stopping selects the point with the best validation metric and restores the corresponding model parameters.

Do not choose the epoch using the test set. Early stopping uses validation data or an internal validation mechanism. A patience setting allows several epochs without improvement before stopping, reducing reactions to a single fluctuation. Look at the trend, not one noisy point.

6. Model capacity: a trade-off, not a contest

Increasing model capacity often improves fit to training data, so training error tends to decrease. Validation error may first decrease and then rise as the model starts describing sample-specific noise. This illustrates the bias–variance trade-off, not a universal law for every model and dataset.

High bias means the model family cannot express the relevant relationship well. High variance means the fitted result changes substantially when the training sample changes. These ideas guide diagnosis, but real decisions require experiments, valid splits and data analysis.

7. What to do: diagnose before choosing a method

If both training and validation loss are high, inspect model expressiveness, features, regularization and optimization convergence. If training loss is low while validation loss is much higher or rising, inspect leakage, noise and distribution differences; then consider a simpler model, regularization, more diverse data, augmentation or early stopping.

Do not apply every method at once. Change one factor at a time, keep the validation protocol fixed and compare results. More data help only when they are relevant and represent future conditions. Excessive regularization can push the model back into underfitting.

8. A small polynomial-model experiment

Compare models with different complexity on one synthetic task. A linear model may be too simple for a wavy relationship; a moderate-degree polynomial may describe it better; a high-degree polynomial may react to noise and individual points. Exact values depend on the random split, so compare training and validation results rather than declaring one degree universally best.

In this example, validation data help choose complexity. After selecting an approach, evaluate it on a separate test set that was not used for tuning. In real work, repeated splits or cross-validation may be useful when the dataset size permits.

Example: compare polynomial models of different complexity

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(7)
X = np.sort(rng.uniform(0, 1, 36)).reshape(-1, 1)
y = np.sin(2 * np.pi * X[:, 0]) + rng.normal(0, 0.18, len(X))

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.25, random_state=7
)

for degree in (1, 4, 12):
    model = make_pipeline(
        PolynomialFeatures(degree=degree, include_bias=False),
        LinearRegression()
    )
    model.fit(X_train, y_train)
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    val_mse = mean_squared_error(y_val, model.predict(X_val))
    print(degree, round(train_mse, 3), round(val_mse, 3))

The columns show polynomial degree, training MSE and validation MSE. As degree increases, training error may decrease, but validation error need not follow. Try another random_state and compare stability. Do not treat validation as the final test.