1.4.5 — Hyperparameters and Model Tuning

1. 1. Parameters and hyperparameters have different roles

Parameters are internal values estimated from training data: linear-model weights, coefficients or tree thresholds. Hyperparameters are set before or around training and control how a model is built: tree depth, regularization strength, number of neighbours, learning rate or number of epochs.

Do not classify a value by its name alone. Ask where it comes from: if optimization estimates it from training data, it is a parameter; if an experimenter or search procedure selects it using validation performance, it is a hyperparameter.

2. 2. What does a hyperparameter control?

Hyperparameters affect the model's available complexity and optimization process. A deeper tree can express more detailed rules but may fit noise. Regularization constrains coefficient magnitude and can reduce variance; too much constraint can cause underfitting.

For a neural network, the learning rate controls optimizer step size: a value that is too large can prevent convergence, while one that is too small can make training slow. Epoch count sets training duration; batch size affects gradient updates and computational load.

3. 3. Build a meaningful search space

A search space defines parameters, allowed values and valid combinations. Do not start with hundreds of arbitrary combinations. Use knowledge of the model, data scale, metric and available resources. Tree depth may use a small integer range; regularization strength is often better explored on a logarithmic scale, such as 0.001 to 100.

Some parameters are conditional: `max_depth` belongs to tree models, while `C` applies to particular scikit-learn SVM and logistic-regression estimators. Do not mix incompatible parameters in one search unless you define separate spaces for each model.

4. 4. Search strategies

Manual tuning is useful for initial diagnostics and testing hypotheses. Grid Search evaluates every specified combination; it is transparent but can become expensive quickly. Random Search samples combinations from specified distributions and can be more efficient when only a few dimensions matter strongly.

Bayesian optimization uses previous trials to select promising next configurations. It can save expensive runs, but needs a sound objective, constraints and evaluation protocol. No method guarantees the global best configuration within a finite budget.

5. 5. Where to tune: train, validation and test

The training set fits model parameters. A validation set or cross-validation compares hyperparameters. The test set stays untouched until the final evaluation of the selected process.

Repeatedly selecting settings using the test set adapts decisions to it, so it is no longer an independent check. With limited data, cross-validation trains on some folds and evaluates on a held-out fold. For temporal or grouped data, folds must respect their structure.

6. 6. Python practice: Pipeline and search

Feature transformations should be fitted inside a Pipeline on each training fold. Otherwise scaling or feature selection may see validation data and cause leakage. The example uses RandomizedSearchCV for classification: the scaler and estimator are combined in a Pipeline, and search uses only training data with internal cross-validation.

Here `C` belongs to logistic regression, so the search space is consistent. In a real project, also record the metric, random_state, number of folds and compute budget.

Code example

from scipy.stats import loguniform
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=3000))
])

search = RandomizedSearchCV(
    pipeline,
    param_distributions={"model__C": loguniform(1e-3, 1e2)},
    n_iter=20,
    scoring="f1",
    cv=5,
    random_state=42,
    n_jobs=-1,
)
search.fit(X_train, y_train)

print("Best hyperparameters:", search.best_params_)
print("Mean CV F1:", search.best_score_)
print("Final test F1:", search.score(X_test, y_test))

7. 7. Tuning is a trade-off, not a magic number

Compare more than the mean metric: fold-to-fold variation, training time, model size, sensitivity to random seeds and performance on important segments. If two configurations differ by less than the uncertainty of the estimate, the simpler and cheaper model may be preferable.

After selection, record the search space, library versions, seed, split strategy, metric and best hyperparameters. Run the final test evaluation once after selection. If the test result leads you to change the model, you need a new independent test or a new experiment.