Correlation, Relationships, and Data Leakage

1. Correlation: how two quantities vary together

Imagine a table containing hourly outdoor temperature and a building’s energy use. If consumption is usually higher during warmer hours, the variables show a positive association. If consumption tends to fall as temperature rises, the association is negative. If the points have no clear direction, the linear relationship may be weak.

Correlation describes co-variation; it does not explain why it happens. It helps us discover patterns, form hypotheses, and spot unusual observations. By itself, however, it does not tell us which variable affects the other, nor does it guarantee that the pattern will persist in new data.

Keep three questions separate: Is there an association? How strong and in what direction is it? Can it support prediction or a causal conclusion? One coefficient cannot answer all three.

2. Pearson and Spearman: two views of association

Pearson’s coefficient measures the direction and strength of a linear relationship. Its value ranges from −1 to +1. The sign indicates direction; the absolute value indicates how closely the points follow a straight-line pattern. +1 represents a perfect positive linear relationship, −1 a perfect negative one. A value near zero indicates little linear association, not the absence of every possible dependency.

Spearman’s coefficient works with ranks — the relative order of values. It measures monotonic association: as X increases, Y generally moves in one direction, even if the rate of change is nonlinear. It can be useful when years of experience tend to increase with tenure, but not by the same amount each year.

Both coefficients have limitations. Outliers can strongly affect Pearson; Spearman is less sensitive to some scale effects, but it is not a universal fix. Ordinal categories, complex relationships, and time series may require additional methods.

Practical rule: plot the data, inspect outliers and distribution shape, then choose a coefficient that matches the question. Do not choose a method merely because it is convenient to calculate.

3. Confounding and causal reasoning

When two variables are associated, several explanations are possible: X affects Y; Y affects X; a third variable affects both; the association is coincidental; or the relationship is more complex than a simple direct effect. A confounder is related to both variables and can create, strengthen, weaken, or even reverse an observed association.

Consider ice-cream sales and sunburn. Both rise in summer as people spend more time outdoors. If season and weather are ignored, we may wrongly attribute an effect to one variable on the other. When exploring data, ask: what else could have produced this pattern?

Causal conclusions need stronger support: a sound study design, clear temporal ordering, a justified set of controlled factors, and tests of alternative explanations. Randomized experiments are often helpful but not always possible. Observational data can also support causal analysis, but only under explicit assumptions and with an appropriate method.

4. Nonlinear relationships and the “zero correlation” trap

Suppose energy use is lowest at a comfortable temperature but rises in both extreme heat, because of air conditioning, and freezing weather, because of heating. The result is a U-shaped relationship. Values behave differently in the middle and at the extremes, so Pearson’s coefficient may be close to zero even though the pattern is clear.

Therefore, “Pearson correlation is zero” means only that there is no strong linear association in the selected sample. It does not mean statistical independence, no pattern, or an unhelpful feature. Use plots, subgroup analysis, and, when needed, models capable of representing nonlinear effects.

The reverse problem also occurs: a strong correlation may be driven by a few extreme points. Check whether the result depends on one entity, one time period, or a handful of erroneous measurements.

5. Correlated features in machine learning

A feature table may contain near-duplicate measurements: temperature in Celsius and Fahrenheit, floor area in two equivalent units, or two sensors installed close to each other. These features carry partly redundant information.

For linear models, strong correlation between predictors can cause multicollinearity: coefficients become unstable and difficult to interpret, and small data changes may noticeably alter estimates. With trees and some ensembles, the effect is different: importance can be shared across similar features, making one feature appear less important than the group as a whole.

Do not automatically remove every correlated feature. First determine whether they duplicate one another, which is more reliable and available, and whether there is a domain reason to keep both. Validate the decision properly and remember that correlations may change in another population or period.

6. Data leakage: the model has seen the answer

Data leakage occurs when training or evaluation uses information that would not be available in a real prediction scenario, or when the evaluation set influences model preparation. The model receives an unfair advantage: the metric looks impressive, but performance collapses on new data.

Imagine predicting a pump failure 24 hours in advance. The table contains temperature, vibration, a maintenance plan, an incident code, and the actual shutdown time. Temperature and vibration may be available before the prediction. The incident code and exact shutdown time appear after the event. Including them does not teach the model to predict failure — it lets the model recognize a failure that has already happened.

Ask this question for every field: could the system have obtained this value at the exact moment when it had to make the prediction? A column’s presence in a historical table does not make it a valid feature.

7. Common leakage types and how to detect them

Target leakage. A feature directly contains the answer or is derived from it. For example, we predict delivery delay but accidentally include the actual delivery date. Trace each feature back to its source and calculation logic.

Temporal leakage. A feature becomes known after the prediction point. This is common in healthcare, finance, industry, and demand forecasting. Record not only the timestamp of an event, but also when its value became available to the system.

Duplicate or group leakage. Copies of a record, measurements from one patient, or data from one device may appear in both train and test. The model effectively recognizes an entity it has already seen. When appropriate, split by patient, customer, building, device, or source document.

Preprocessing leakage. If normalization, imputation, feature selection, or dimensionality reduction is performed on the full table before splitting, test-set statistics influence training. Every transformation that learns something from data must be fitted on train only, inside a pipeline.

There is also leakage during model selection: if you repeatedly inspect test results and change the model accordingly, the test set gradually becomes part of development. Use validation or cross-validation for selection; reserve test for a final independent evaluation.

8. Build an honest model evaluation

1. Describe the task. What is being predicted, for which entity, and over what horizon?
2. Fix the prediction point. When exactly must the system produce an answer?
3. Audit feature availability. Record each field’s source, availability time, and possible delay.
4. Choose a suitable split. Random splitting is appropriate only when observations are independent and it matches deployment; use time splits for future forecasting and group splits for unseen entities.
5. Fit transformations on train only. Use a pipeline so imputation, scaling, and feature selection cannot see validation or test.
6. Tune on validation. Do not choose parameters based on the test metric.
7. Evaluate the frozen solution on test once. Save the data version, split rules, and evaluation limitations.

9. Practice: audit a failure-prediction model

A company wants to predict a fan failure one day ahead. The table contains motor temperature, vibration, load, last maintenance date, registered incident code, actual shutdown time, and final repair status.

Complete the audit:

1. Write down the prediction point — for example, every day at 09:00 — and the 24-hour horizon.
2. For every field, decide whether it is available at 09:00. Check whether maintenance dates can be recorded retrospectively.
3. Decide whether the data must be split by time or by fan identifier.
4. Describe where imputation and scaling will be fitted.
5. Explain why a high correlation between the incident code and target does not make that code a valid feature.

A strong answer is more than a list of fields to keep or remove. It explains the prediction point, data provenance, and how the experiment simulates real deployment.