Learning outcomes

After this lesson, you will be able to:

Key questionA model may answer almost perfectly on examples it has already seen and still fail on new objects. Its practical value depends not on remembering the training sample, but on transferring a learned pattern reliably to data encountered in real use.

1. What does generalization mean?

Generalization is the ability of a trained model to maintain acceptable performance on new examples that were not used to fit its parameters. “New” matters: the model may see different rows, images, documents or events, while the task and conditions should remain sufficiently comparable to the intended use.

Imagine a model estimating pump failure risk from temperature, vibration and pressure. Strong results on the measurements used for training do not show that it will recognize failures on another pump, in another factory or next season. Generalization concerns transfer of a pattern, not memorization of records.

Empirical riskR̂emp(f) = (1 / n) Σi=1n L(f(xi), yi)
Training and generalizationTraining examples pass through Model; then New examples is evaluated. Transfer estimate.Training examplesModelNew examplesTransfer estimate
Training and generalization — Transfer estimate

2. From sample error to expected risk

In lesson 1.4.2 we studied a loss function and the average loss on training examples. On a finite sample this is empirical risk. We are also interested in expected loss over the unknown distribution of future examples.

Let P denote the target environment's distribution, f the model and L the loss function. Expected risk averages loss over possible input–target pairs according to P. We cannot enumerate every future case, so we estimate risk on a finite sample and try to make that sample independent and representative.

Expected riskRP(f) = E(x,y)∼P [L(f(x), y)]
A valid splitTraining pass through Validation; then Final test is evaluated. No overlap.TrainingValidationFinal testNo overlap
A valid split — No overlap

3. Reading the generalization gap

A practical starting point is to compare the same metric on the training set and an independent test set. If test error is substantially higher, there is a generalization gap. It may indicate overfitting, but it does not identify the cause by itself: sample differences, a small test set, leakage or a flawed split may also be responsible.

If both errors are high, the model may lack capacity, features may be weak, data may be noisy or optimization may be incomplete. If both are low, that is encouraging only when the experiment is valid and the test sample matches the target use. A number without context is not proof.

Generalization gapG = R̂test(f) − R̂train(f)
Data leakageFull dataset pass through Before split; then Test → train leakage is evaluated. Invalid estimate.Full datasetBefore splitTest → train leakageInvalid estimate
Data leakage — Invalid estimate

4. The sample should reflect future use

A random split is not always appropriate. In classification with a rare class, stratification can preserve class proportions across subsets. If several rows belong to one patient, customer, device or object, split by group; otherwise the model may encounter nearly identical records in both training and test sets.

For forecasting future values in a time series, do not randomly shuffle past and future. Train on an earlier period and evaluate on a later one. The split should imitate deployment, not merely produce convenient numbers.

Distribution shiftPtrain(X,Y) ≠ Pdeploy(X,Y)
Environment shiftTraining environment pass through Deployment environment; then Distributions differ is evaluated. Re-evaluate.Training environmentDeployment environmentDistributions differ · Re-evaluate
Environment shift — Re-evaluate

5. Data leakage: when the test is no longer independent

Leakage occurs when information unavailable at prediction time enters training or data preparation. For example, computing scaling statistics on the full dataset before splitting allows test data to influence the training transformation. Another example is a feature created only after the event that the model is supposed to predict.

Protection: split first, then fit transformations only on the training portion. A Pipeline can combine preprocessing and the model safely. Also inspect timestamps, duplicates, related groups, feature sources and when each feature becomes available. Even hidden leakage can make a score implausibly high.

6. When the environment itself changes

Even a sound test score refers to a particular data distribution. In production, that distribution may change. Feature shift changes the input distribution; label or prior shift changes target frequencies; concept shift changes the relationship between features and outcomes. Equipment may be replaced, a sensor recalibrated or operating conditions altered.

Generalization therefore does not mean “the model works everywhere and forever.” A test score addresses similar conditions; transfer to a new environment requires separate validation. Monitor inputs, quality when labels arrive, errors by segment and predefined model-review triggers.

7. What a test score can and cannot prove

An independent test provides a useful estimate, but it has statistical uncertainty. On a small sample, a few difficult cases can change the result substantially. With imbalanced classes, accuracy may look high while an important rare class is missed. Choose metrics according to the task and cost of errors; where appropriate, report sample size, uncertainty intervals or stability analysis.

Do not repeatedly use the test set to choose features, architectures or thresholds: it gradually becomes another validation set. Use validation data or cross-validation for model selection, and reserve the final test for an independent assessment. Hyperparameter tuning is covered in lesson 1.4.5; cross-validation and stability appear in a later unit.

8. A practical evaluation protocol

Before trusting a score, define the target environment and prediction time; identify the independent unit; choose a split that prevents related objects from crossing partitions; decide the metric before inspecting test results; fit every transformation inside the training pipeline; compare against a simple baseline; inspect errors across important segments; and record settings and the experiment's random seed.

Keep evaluating after deployment. Data distributions and model quality can change. The system needs a clear process for detecting degradation, obtaining new labels, retraining and rolling back. Generalization is not a permanent property: it is performance supported for specific data, time and conditions.

Practice task: design an honest model evaluation

Scenario. You are building a model to predict equipment failure from sensor measurements. The table contains several rows per device, measurements span two years, and failures are recorded later.

  1. Define the independent observation: a measurement row or a device.
  2. Propose a split for predicting failures on unseen devices. Then propose a split for forecasting next month for devices already known.
  3. Identify at least three possible leakage sources and explain why they are invalid.
  4. Choose a metric and a baseline. Explain why accuracy alone may be insufficient when failures are rare.
  5. Specify two signals to monitor after deployment.

A strong answer: the evaluation imitates the real use case, prevents related records from crossing partitions and does not use future information.

Example: a leakage-safe classification pipeline

# Python 3.10+; requires pandas and scikit-learn
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import balanced_accuracy_score, classification_report

# target is the class; features are available at prediction time.
df = pd.read_csv("equipment.csv")
X = df[["temperature", "vibration", "pressure"]]
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# Pipeline fits StandardScaler only on X_train.
model = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=1000, class_weight="balanced")
)
model.fit(X_train, y_train)

pred = model.predict(X_test)
print("Balanced accuracy:", balanced_accuracy_score(y_test, pred))
print(classification_report(y_test, pred))

# Do not repeatedly tune the model or thresholds on this test set.
# Grouped/time data requires a different split protocol.

Sources and further reading

  1. Scikit-learn: Common pitfalls and recommended practices
  2. Scikit-learn: Cross-validation
  3. Goodfellow, Bengio & Courville — Deep Learning, Chapter 5
  4. James et al. — An Introduction to Statistical Learning, 2nd ed.
  5. Quionero-Candela et al. — Dataset Shift in Machine Learning

Lesson glossary

GeneralizationThe ability to maintain model performance on new data from the target environment.
Expected riskExpected average model loss under the target example distribution.
Empirical riskAverage loss on a finite observed sample.
Generalization gapDifference between an estimate of error on new data and training error under a comparable metric.
RepresentativenessHow well a sample reflects relevant properties of the target population and use case.
Data leakageUse of information that should not be available during training or prediction.
Distribution shiftA change in data statistics between training and deployment.
BaselineA simple reference model or rule used for comparison.
StratificationA splitting method that preserves specified class proportions.
MonitoringObservation of model inputs and performance after deployment.

Knowledge check

Choose one answer. An explanation appears after checking.

Question 1

Which statement best describes generalization?

Question 2

What is expected risk?

Question 3

Test error is much higher than training error. What is the sound conclusion?

Question 4

Several records belong to one patient. How should the data be split?

Question 5

Why can scaling before the split cause leakage?

Question 6

How should a time series be split to forecast the future?

Question 7

Can you repeatedly select a model using the test set?

Question 8

Does the previous test score guarantee performance after the environment changes?