Training, Validation, and Test Data

1. Why split data at all?

A model must work not only on rows it has already seen. We need to estimate how it will behave on new observations that arrive after deployment. This is a test of generalization.

Measuring performance on training examples is usually optimistic because the model has already adapted to them. A split simulates the boundary between the past available during development and the future that is not yet known.

2. Three sets, three different decisions

Train is used to fit model parameters and learnable data transformations.

Validation supports development choices: model comparison, hyperparameter selection, feature decisions, classification thresholds, and other tuning.

Test is an independent final assessment of the selected process. It estimates expected performance; it is not another tuning round.

3. Why you must not keep checking the test set

Whenever you change a model after seeing its test metric, the test set indirectly participates in development. Even if the optimizer never fits on test rows, a person has used their result to select a solution.

This creates adaptive overfitting: among many candidates, we gradually select the one that happened to perform well on this particular test sample. The final metric becomes optimistically biased. If the test set has already been used repeatedly, collect a new independent test set or explicitly report the limitation.

4. Random and stratified splitting

Random splitting is suitable when observations are independent and similarly distributed, and row order has no meaning. In imbalanced classification, a random split can produce very different class proportions across partitions.

Stratification preserves approximate class proportions in train, validation, and test. It is useful for rare classes, but it does not remove dependence between rows, time leakage, or duplicates. Stratification is not a universal guarantee of a fair evaluation.

5. Group splitting: prevent the model from recognizing entities

Imagine a medical imaging dataset with dozens of images per patient. If some images from a patient enter train and others enter test, the model may recognize patient-specific characteristics. The metric then answers “can we recognize familiar patients?” rather than “can we work with new people?”

Group splitting keeps all rows from one group in a single partition. A group can be a patient, customer, building, device, production batch, or source document. A group identifier may be needed for splitting even when it should not be fed to the model as an ordinary feature.

6. Time-based splitting: the past predicts the future

For demand, energy, failures, or price forecasting, random shuffling is often wrong. It can place later observations in train and earlier observations in test, allowing the model to learn from the future.

Use chronological boundaries: earlier periods for train, the next period for validation, and the latest period for test. If the model is retrained on a rolling window, evaluate several successive time windows. Account for forecast horizon, data publication delays, and seasonality.

7. Data leakage: information crossed the boundary

Leakage occurs when fitting or selection uses information unavailable in the real prediction scenario or information that should have remained hidden for independent assessment. Common causes include duplicates across partitions; the same entity in train and test when testing new entities; shuffled time series; features created after the target event; and scaling or feature selection before splitting.

Feature meaning can make leakage especially subtle. For example, we forecast tomorrow’s energy use but include tomorrow’s actual building occupancy. The value exists in the table, but it is not known at prediction time.

8. Preprocessing is part of the model, not neutral housekeeping

Mean imputation, scaling, feature selection, PCA, category vocabularies, and target encoding learn parameters from data. Fit them on train only, then apply the learned transformations to validation and test without refitting.

The correct pipeline is: split; fit transformations on the training partition; transform the others; fit the model; compare on validation. During cross-validation, the entire pipeline is refitted inside each fold. This prevents validation statistics from leaking into training.

9. Cross-validation and hyperparameter tuning

With limited data, a single validation split can be unstable. K-fold cross-validation divides the training portion into K folds; each fold is used once for validation while the others train the model. The mean and spread of the metric show how sensitive the result is to the split.

Use StratifiedKFold for many classification tasks, GroupKFold for dependent entities, and TimeSeriesSplit for temporal data. Nested cross-validation can estimate the full hyperparameter-selection process when no additional independent set is available. The test set remains untouched.

10. A protocol for honest evaluation

Before training, document the target and forecast horizon; the independent unit; groups and time boundaries; features available at prediction time; metric and baseline; missing-value rules; and tuning procedure. After selecting a solution, freeze the pipeline and evaluate it once on test.

Save the random seed, data version, split code, metrics for important segments, and confidence intervals or fold-to-fold variation. A single number without context does not describe model reliability.

Example: group split and Pipeline

The example illustrates the principle: grouping happens before fitting, while imputation and scaling live inside the Pipeline. It assumes that column names and feature lists have already been prepared.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import GroupShuffleSplit
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

# Separate the target and grouping variable first.
X, y, groups = data[features], data["target"], data["patient_id"]

splitter = GroupShuffleSplit(
    n_splits=1, test_size=0.2, random_state=42
)
train_idx, test_idx = next(splitter.split(X, y, groups))

numeric = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
categorical = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore")),
])
preprocess = ColumnTransformer([
    ("num", numeric, numeric_features),
    ("cat", categorical, categorical_features),
])
model = Pipeline([
    ("preprocess", preprocess),
    ("classifier", LogisticRegression(max_iter=1000)),
])
model.fit(X.iloc[train_idx], y.iloc[train_idx])
# Use test only after the solution is frozen.

For a temporal task, use a chronological split instead; for imbalanced classification, consider stratification without breaking group or time constraints.

Practical assignment

Practice: predict equipment failure. The table contains device_id, timestamp, temperature, vibration, maintenance_flag, and failure_next_24h. Decide which features are available 24 hours before prediction. Design two scenarios: forecasting for known devices and for unseen devices. For each, choose a split, define train/validation/test, and explain where the imputer and scaler are fitted.

1. State the independent unit and deployment scenario.
2. Mark features that are not yet known at prediction time.
3. Choose and justify a splitting strategy.
4. Describe the fit/transform order and when final testing occurs.