1.4.2 — Model Training and the Loss Function

1. From a model to training

In 1.4.1, a model was represented as ŷ = f(x; θ). Here θ denotes parameters, x the input features, and ŷ the prediction. While θ is fixed, the model simply applies its rule to new objects. Training asks a different question: how should we choose θ so predictions on known training examples agree with their target values y?

Imagine a model estimating equipment repair time. For each completed job, we know features—equipment type, age, number of failures—and the actual repair time. The model produces a prediction. If predictions systematically differ from the observed values, parameters can be changed. But “change them to improve the result” must be made precise: what counts as an error, and how do we compare parameter settings?

2. Error and the loss function

For one example, we can compare prediction ŷᵢ with the known target yᵢ. The difference shows the direction of the error, but positive and negative differences can cancel when summed. Therefore, we often use the absolute difference or its square rather than the raw signed difference.

A loss function defines how to assign a numerical penalty to a mismatch between a prediction and its target. It turns the quality of one prediction into a number. The value depends on the task and the chosen function: the same deviation can have different costs under different problem definitions.

For regression, squared error is common. Squaring makes the contribution non-negative and penalizes large deviations more strongly. An error of 4 contributes 16, while an error of 1 contributes 1. This is useful when large misses are especially undesirable, but it also makes the method sensitive to outliers.

3. From one error to the training objective

A training set contains n examples. The loss function computes a value ℓᵢ for each example. The average empirical loss is the mean of these values. Training is often formulated as finding parameters that minimize this average.

Important: we minimize a chosen mathematical objective on training data; we do not prove that the model will be error-free in the real world. The result depends on how well the training data represent future situations, which features are available, and whether the loss function reflects the actual consequences of errors.

Do not confuse loss with an evaluation metric. Loss is the criterion used during training to update parameters. Model evaluation metrics and validation protocols are covered in theme 1.5. A low training loss alone does not guarantee good performance on unseen data.

4. Finding a minimum: gradient descent

If we can slightly change parameters and calculate how the loss changes, we can move in a direction that reduces it. A derivative describes the local rate of change with respect to a parameter. The gradient combines these derivatives for all parameters and points toward the direction of steepest local increase. To reduce the objective, we step in the opposite direction.

At each step, the algorithm calculates the gradient of the current loss and updates the parameters. The step size is set by the learning rate. A step that is too large may jump over a minimum or diverge; one that is too small makes training slow. Detailed tuning of learning rate, batch size, and other hyperparameters belongs in lesson 1.4.5.

For linear regression with MSE, the objective is convex under the standard setup: there are no multiple local minima, and gradient methods can reach a global minimum under suitable conditions. For complex neural networks, the loss surface is usually non-convex, so optimization is more complicated.

5. Worked example: a linear model

Suppose a model predicts energy consumption from equipment operating time: ŷ = wx + b. We have four toy observations. Start with w = 0 and b = 0. The model initially predicts 0 for every example. We calculate MSE, then the gradients with respect to w and b, and update the parameters.

The gradient with respect to w tells us how the mean squared error changes when the line's slope changes; the gradient with respect to b tells us how it changes when the line shifts. After an update, parameters change, so predictions and loss are recalculated. Repeating the steps gradually selects a line that better fits the training points.

The code below implements simple batch gradient descent without third-party libraries. Each iteration uses all four observations. It is an educational implementation to understand the mechanism, not a replacement for a production-grade library.

x_values = [1.0, 2.0, 3.0, 4.0]
y_values = [3.0, 5.0, 7.0, 9.0]

w = 0.0
b = 0.0
learning_rate = 0.01
epochs = 2000
n = len(x_values)

for epoch in range(epochs):
    predictions = [w * x + b for x in x_values]
    errors = [pred - target for pred, target in zip(predictions, y_values)]
    mse = sum(error ** 2 for error in errors) / n

    grad_w = (2 / n) * sum(error * x for error, x in zip(errors, x_values))
    grad_b = (2 / n) * sum(errors)

    w -= learning_rate * grad_w
    b -= learning_rate * grad_b

print(f"w = {w:.3f}, b = {b:.3f}")
print(f"MSE = {mse:.6f}")
print("predictions =", [round(w * x + b, 2) for x in x_values])

Expected result: w approaches 2, b approaches 1, and MSE approaches zero. A small difference remains because the number of iterations is finite.

Walkthrough: compute predictions; calculate errors = prediction − target; compute MSE and derivatives with respect to w and b; then update parameters. The gradient is calculated before the update, while the next iteration's loss reflects the new parameters.

6. Other losses and limitations

MSE is not universal. MAE uses absolute deviations and is less affected by individual large errors, but it has a kink at zero. Huber loss combines a quadratic region for small errors with a linear region for large ones. Classification often uses cross-entropy, which compares predicted probabilities with the true label.

Choosing a loss is a substantive decision, not a cosmetic setting. If an error of 10 units is twice as costly as an error of 5, a quadratic penalty may be justified. If rare outliers are measurement mistakes, MSE may overfit them. If underestimation and overestimation have different consequences, a symmetric loss may be inappropriate.

Optimization also has limitations: sensitivity to feature scales, initialization, noise, and missing values; possible overfitting to training examples; and dependence on the algorithm and hyperparameters. Overfitting and underfitting are covered in 1.4.3, generalization in 1.4.4, and tuning in 1.4.5.

7. What to remember

Training connects data, a model, a loss function, and an optimizer. Data provide examples; the model produces predictions; loss measures the mismatch; and the optimizer changes parameters. These roles are distinct, and understanding each is necessary for analyzing model behavior.

A minimum training loss is the objective of a chosen procedure, not a guarantee of fairness, causal validity, robustness, or good performance on new data. Always ask: what data were used, what does the loss penalize, which errors remain invisible, and how was the result evaluated?